ZjDSrSSKrSSKJr SSKrSSKJr SSKJr SSK J r J r SSK Jr SSKJrJr S S KJrJrJrJrJrJrJrJrJr S S KJr \R<"\5r \"S S9\"SS\ 555r!"SS\5r""SS\5r#"SS\5r$"SS\5r%"SS\5r&"SS\5r'\"SS\55r(\ r)"SS \(\5r*\"S!S9"S"S#\(55r+"S$S%\5r,"S&S'\5r-/S(Qr.g))zPyTorch UniSpeech model.N) dataclass)initialization) ModelOutputWav2Vec2BaseModelOutput)PreTrainedModel)auto_docstringlogging) Wav2Vec2EncoderWav2Vec2EncoderStableLayerNormWav2Vec2FeatureEncoderWav2Vec2FeatureProjectionWav2Vec2ForCTC!Wav2Vec2ForSequenceClassificationWav2Vec2GumbelVectorQuantizer Wav2Vec2ModelWav2Vec2PositionalConvEmbedding)UniSpeechConfigzh Output type of [`UniSpeechForPreTrainingOutput`], with potential hidden states and attentions. ) custom_introc\rSrSr%SrSr\RS-\S'Sr \RS-\S'Sr \RS-\S'Sr \RS-\S'Sr \ \RS-\S'Sr\ \RS-\S 'S rg) UniSpeechForPreTrainingOutput+a loss (*optional*, returned when model is in train mode, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the contrastive loss (L_m) and the diversity loss (L_d) as stated in the [official paper](https://huggingface.co/papers/2006.11477). projected_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`): Hidden-states of the model projected to *config.proj_codevector_dim* that can be used to predict the masked projected quantized states. projected_quantized_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`): Quantized extracted feature vectors projected to *config.proj_codevector_dim* representing the positive target vectors for contrastive loss. codevector_perplexity (`torch.FloatTensor` of shape `(1,)`): The perplexity of the codevector distribution, used to measure the diversity of the codebook. Nlossprojected_statesprojected_quantized_statescodevector_perplexity hidden_states attentions)__name__ __module__ __qualname____firstlineno____doc__rtorch FloatTensor__annotations__rrrrtupler __static_attributes__r!ڀ/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/unispeech/modular_unispeech.pyrr+s &*D%  d ")15e''$.5;? 1 1D 8?6:5,,t3:59M5**+d2926Je''(4/6r,rc\rSrSrSrg) UniSpeechPositionalConvEmbeddingHr!Nr"r#r$r%r+r!r,r-r/r/Hr,r/c\rSrSrSrg)UniSpeechFeatureEncoderLr!Nr1r!r,r-r4r4Lr2r,r4c\rSrSrSrg)UniSpeechFeatureProjectionPr!Nr1r!r,r-r7r7Pr2r,r7c\rSrSrSrg)UniSpeechEncoderTr!Nr1r!r,r-r:r:Tr2r,r:c\rSrSrSrg)UniSpeechEncoderStableLayerNormXr!Nr1r!r,r-r=r=Xr2r,r=c*\rSrSr\S5rSrSrg)UniSpeechGumbelVectorQuantizer\cURSS9n[R"[R"[R"X5SS9*5R5nU$)Nrdim)meanr'expsumxlogy)probsmarginal_probs perplexitys r-_compute_perplexity2UniSpeechGumbelVectorQuantizer._compute_perplexity]sG*YY %++n*U[] ^^_cce r,cURup#nURU5nURX#-UR-S5nUR(a[ R RUR5URSS9RU5n[R"URX#-URS5R5SS9nURU5nOyURSS9nUR"UR6R!SURSS5S5nURX#-URS5nURU5nURX#-S5nUR#S5UR$-n U RX#-URUR&S5n U R)S5RX#S5n X4$)NrET)tauhardrCrg?)shape weight_projview num_groupstrainingnn functionalgumbel_softmaxfloat temperaturetype_asr'softmaxrMargmax new_zerosscatter_ unsqueeze codevectorsnum_varsrH) selfr batch_sizesequence_length hidden_sizecodevector_probscodevector_soft_distrLcodevector_idxcodevectors_per_grouprcs r-forward&UniSpeechGumbelVectorQuantizer.forwardcs3@3F3F0 [((7 %**:+G$//+Y[]^ ==!}};;##%4+;+;$ < gm$  $)=="":#?RTU[[]ce$ 112FGJ+11b19N,66 8K8KLUUN''A.   044Z5QSWSbSbdfg 112BCJ+001MrR 0 : :2 >AQAQ Q+001Mt`d`m`moqr !oob)..zBO &&r,r!N)r"r#r$r% staticmethodrMrmr+r!r,r-r@r@\s #'r,r@c\rSrSr%\\S'SrSrSrSr Sr Sr Sr \ R"5S5rS\ R \-4S jrS \S \ R 4S jrS rg)UniSpeechPreTrainedModelconfig unispeech input_valuesaudioTc [U[5(au[R"URR SSS9 [R "URR5 [R"UR5 g[U[5(a[R"URR SS[R"SURRSURR-- 5-S9 [R "URRS5 g[U["5(a[R"SUR$R&- 5n[R"UR$R U*US9 [R"UR$RU*US9 g[U[(R*5(ac[R"UR SUR,R.S9 URb![R "UR5 gg[U[(R0[(R245(aA[R "UR5 [R4"UR 5 g[U[(R65(a[R8"UR 5 URb_[R"UR:URURS-- 5n[R"URU*US9 ggg)zInitialize the weightsr)rFstdrr )abN) isinstancer@initnormal_rTweightzeros_biasuniform_rcr/convmathsqrt kernel_size in_channels constant_r7 projection in_featuresrXLinearrsinitializer_range LayerNorm GroupNormones_Conv1dkaiming_normal_groups)remoduleks r- _init_weights&UniSpeechPreTrainedModel._init_weightss< f< = = LL++22! D KK**// 0 MM&,, -  @ A A LL "" !v{{'>'>q'AFKKD[D['["\]]  NN6;;++Q /  : ; ; !f//;;; MM&++00QB! <  * * LLSdkk6S6S T{{& FKK(' r|| < = = KK $ JJv}} %  * *   /{{&IIfmmv/A/AFDVDVWXDY/YZ[ fkkaR15'+r, input_lengthscSn[URRURR5Hup4U"XU5nM U$)z8 Computes the output length of the convolutional layers c8[R"X- USS9S-$)Nfloor) rounding_moder)r'div) input_lengthrstrides r-_conv_out_lengthSUniSpeechPreTrainedModel._get_feat_extract_output_lengths.._conv_out_lengths 99\7wWZ[[ [r,)ziprs conv_kernel conv_stride)rerrrrs r- _get_feat_extract_output_lengths9UniSpeechPreTrainedModel._get_feat_extract_output_lengthssG  \ $'t{{'>'> @W@W#X K,]PM$Yr,feature_vector_lengthattention_maskcURSS9SS2S4nURU5R[R5nUR Sn[R "XQ4URURS9nSU[R"UR SURS9US- 4'URS/5RS5RS/5R5nU$)NrErCr)dtypedevicer)r) cumsumrtor'longrSzerosrrarangeflipbool)rerrnon_padded_lengthsoutput_lengthsrfs r-"_get_feature_vector_attention_mask;UniSpeechPreTrainedModel._get_feature_vector_attention_masks,22r2:1b5A>>?QRUUV[V`V`a#))!,   /~7K7KTbTiTi uv ^%9%9!%<^EZEZ[]kno]opq',,bT299"=BBB4HMMOr,r!N)r"r#r$r%rr)base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attnr'no_gradr LongTensorintrrr+r!r,r-rqrqs{ #$O&*#N ]]_66B e>N>NQT>T   ]b]m]m r,rqc\rSrSrS\4SjrSrSS\RS-S\RS-S\RS-S \ S-S \ S-S \ S-S \ \ -4S jjr Srg)UniSpeechModelrsc[RX5 Xl[U5Ul[ U5UlURS:dURS:aG[R"[R"UR5R55UlUR (a[#U5UlO['U5UlUR)5 g)Nrx)rq__init__rsr4feature_extractorr7feature_projectionmask_time_probmask_feature_probrX Parameterr'Tensorrhrmasked_spec_embeddo_stable_layer_normr=encoderr: post_init)rerss r-rUniSpeechModel.__init__s ))$7 !8!@"[TU]U5 [U5Ul[R "UR 5Ul[U5Ul [R"URUR5Ul [R"URUR5Ul[R"URUR 5Ul[R "UR$5UlUR)5 g)N)superrrrtrXDropoutfeat_quantizer_dropoutdropout_featuresr@ quantizerrcodevector_dimproj_codevector_dim project_qrh project_hidnum_ctc_classesctc_proj final_dropoutdropoutr)rers __class__s r-r UniSpeechForPreTraining.__init__)s  '/ " 6+H+H I7?6#8#8&:T:TU99V%?%?ASAST &"4"4f6L6LM zz&"6"67  r,r\c$XRlg)zR Set the Gumbel softmax temperature to a given value. Only necessary for training N)rr\)rer\s r-set_gumbel_temperature.UniSpeechForPreTraining.set_gumbel_temperature8s&1"r,cLURRR5 g)z Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training. N)rtr_freeze_parametersrs r-r.UniSpeechForPreTraining.freeze_feature_encoder>s ((;;=r,target_featuresnegative_featurespredicted_featuresc[R"X/SS9n[R"UR5UR5SS9nUR U5nXC- nU$)z Compute logits for contrastive loss based using cosine similarity as the distance measure between `[positive_feature, negative_features]` and `[predicted_features]`. Additionally, temperature can be applied. rrCrE)r'catcosine_similarityr[r])rrrr\logitss r-compute_contrastive_logits2UniSpeechForPreTraining.compute_contrastive_logitsEs\ ))_$HaP(();)A)A)C_EZEZE\bde0% r,Nrurrrrrc UbUOURRnURUUUUUS9nUSnURUS5n UR U 5upUR U R UR RR55n URU 5n [R"URS5URS55RURR5n U RSS5n [R "U 5R#5R UR$5n U RSS5n U R'S5n UR)U S5U R)U )S5-nUR+U5nUR-U5nSnU(dUb XX4USS-$XU 4USS-$[/UUU U UR0UR2S9$) a[ Example: ```python >>> import torch >>> from transformers import AutoFeatureExtractor, UniSpeechForPreTraining >>> feature_extractor = AutoFeatureExtractor.from_pretrained("microsoft/unispeech-large-1500h-cv") >>> model = UniSpeechForPreTraining.from_pretrained("microsoft/unispeech-large-1500h-cv") >>> # TODO: Add full pretraining example ```NrrrrErxr )rrrrrr )rsrrtrrrrrrrr'emptysizefill_ replace_probr bernoullirrrb masked_fillrrrrr )rerurrrrroutputstransformer_featuresrquantized_featuresrprob_replace_matrixsampled_replace_matrixrrs r-rmUniSpeechForPreTraining.forwardYs,&1%,A,A$..BWBWB]B],^_!--.@A#kk*>*C*CA*FH\HaHabcHdekk KK $ $ 2;;AqA!&1D!E!J!J!L!O!OPdPkPk!l!7!A!A!Q!G!7!A!A"!E%112H#N  * *,B+BC H  f%v&4F^ahijikalll(>STW^_`_aWbb b,1'9"7!//))   r,)rrrrrrrt)r)NNNN)r"r#r$r%rrrrrror'r(rr rrr*rrmr+ __classcell__)rs@r-rr#s  1#1 >  ** ,,"-- &/3)-,0#' E llT)E  t+E  $; E #Tk E D[ E  . .E E r,rc\rSrSrSrg)UniSpeechForCTCir!Nr1r!r,r-rrr2r,rc\rSrSrSrg)"UniSpeechForSequenceClassificationir!Nr1r!r,r-rrr2r,r)rrrrrq)/r&r dataclassesrr'torch.nnrXrr}modeling_outputsrrmodeling_utilsrutilsr r wav2vec2.modeling_wav2vec2r r rrrrrrrconfiguration_unispeechr get_loggerr"loggerrr/r4r7r:r=r@rqrrrrr__all__r!r,r-r"sH ! &D-,   5   H %  7K7  7. 'F  4  !:    &D *'%B*'ZHHHV3H -}H V w 6w  w t n  )J  r,