ZjxgSrSSKrSSKrSSKJr SSKJr SSKrSSK r SSK J r SSK J r SSKJr SS KJr SS KJr SS KJr SS KJr SS KJr SSKJr SSKJr SSKJ r J!r!J"r"J#r#J$r$J%r% SSK&J'r'J(r(J)r) SSK*J+r+ SSK,J-r-J.r.J/r/J0r0J1r1J2r2J3r3 SSK4J5r5 Sr6Sr7\3Rp"\95r:Sr;\/"SS9\"SS\-555r<SiS\=\>\>4S\?S\>S \ RS-S!\>S"\R4 S#jjrBSjS$\=S%\>S&\RS-4S'jjrC"S(S)\5rD"S*S+\5rE"S,S-\5rF"S.S/\ R5rH"S0S1\ R5rI"S2S3\ R5rJ"S4S5\ R5rKSkS6\ RS7\ RS8\ RS9\ RS \ RS-S:\?S-S;\?S<\+\.4S=jjrM"S>S?\ R5rN"S@SA\ R5rO"SBSC\5rP"SDSE\5rQ"SFSG\ R5rR"SHSI\ R5rS"SJSK\ R5rT"SLSM\ R5rU"SNSO\ R5rV"SPSQ\ R5rW\/"SRSS\(55rX\/"STSU\X55rY\/"SVS9"SWSX\X55rZ\/"SYS9"SZS[\X55r[\/"S\S9"S]S^\X55r\\/"S_S`\X55r]"SaSb\ R5r^"ScSd\ R5r_\/"SeS9"SfSg\X55r`/ShQrag)lzPyTorch Wav2Vec2 model.N)Callable) dataclass) load_file)nn)CrossEntropyLoss)initialization)ACT2FN)is_deepspeed_zero3_enabled)is_fsdp_managed_module)create_bidirectional_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputCausalLMOutputSequenceClassifierOutputTokenClassifierOutputWav2Vec2BaseModelOutput XVectorOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel*get_torch_context_manager_or_global_device)Unpack) ModelOutputTransformersKwargsauto_docstring cached_filecheck_torch_load_is_safeis_peft_availablelogging)Wav2Vec2Configzadapter.{}.binzadapter.{}.safetensorsza Output type of [`Wav2Vec2ForPreTraining`], with potential hidden states and attentions. ) custom_introcf\rSrSr%SrSr\RS-\S'Sr \RS-\S'Sr \RS-\S'Sr \RS-\S'Sr \ \RS-\S'Sr\ \RS-\S 'Sr\RS-\S 'Sr\RS-\S 'S rg) Wav2Vec2ForPreTrainingOutputBa loss (*optional*, returned when `sample_negative_indices` are passed, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the contrastive loss (L_m) and the diversity loss (L_d) as stated in the [official paper](https://huggingface.co/papers/2006.11477). projected_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`): Hidden-states of the model projected to *config.proj_codevector_dim* that can be used to predict the masked projected quantized states. projected_quantized_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`): Quantized extracted feature vectors projected to *config.proj_codevector_dim* representing the positive target vectors for contrastive loss. codevector_perplexity (`torch.FloatTensor` of shape `(1,)`): The perplexity of the codevector distribution, used to measure the diversity of the codebook. contrastive_loss (*optional*, returned when `sample_negative_indices` are passed, `torch.FloatTensor` of shape `(1,)`): The contrastive loss (L_m) as stated in the [official paper](https://huggingface.co/papers/2006.11477). diversity_loss (*optional*, returned when `sample_negative_indices` are passed, `torch.FloatTensor` of shape `(1,)`): The diversity loss (L_d) as stated in the [official paper](https://huggingface.co/papers/2006.11477). Nlossprojected_statesprojected_quantized_statescodevector_perplexity hidden_states attentionscontrastive_lossdiversity_loss)__name__ __module__ __qualname____firstlineno____doc__r(torch FloatTensor__annotations__r)r*r+r,tupler-r.r/__static_attributes__r0/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.pyr&r&Bs$&*D%  d ")15e''$.5;? 1 1D 8?6:5,,t3:59M5**+d2926Je''(4/615e''$.5/3NE%%,3r;r&shape mask_prob mask_lengthattention_mask min_masksreturnc ^^^^^UunmTS:a [S5eTT:a[STSTS35e[RRS5R 5mUUUUU4SjnUb-UR 5R S5R5O[U5Vs/sHnTPM snn[R"UT4[S 9n /n U"T5n U S :XaU $UHn U"U 5n [RR[R"U TS- - 5U S S 9n[U5S :XaTS- nOUS n[R"U[R"X- [R S 9U-/5nU R#U5 M [R$"U 5n [R&"U SS2SS2S4X[T45n U R)X[T-5n [R"T5SSSS24n[R&"UX[T45R)X[T-5nU U-n U R+5TS- :a TS- XTS- :'[R,"XSS5 U $s snf) a2 Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://huggingface.co/papers/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. r!z&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `c>[TU-T- T-5n[UT5nUT-T:aTT-nUTS- - U:a[UTS- - S5nU$)z;Given input length, compute how many spans should be maskedr!r)intmax) input_lengthnum_masked_spanepsilonr?r>rAsequence_lengths r<compute_num_masked_span6_compute_mask_indices..compute_num_masked_spansoi,6DwNOoy9 [ (? :-r?r@rA batch_sizerL_ input_lengths spec_aug_maskspec_aug_mask_idxsmax_num_masked_spanrHrIspec_aug_mask_idxdummy_mask_idxoffsetsrJrKs `` ` @@r<_compute_mask_indicesrqes0#(JQABB_$]^i]j&&7q :  iinnQ$$&G  $  % ##B'..0',Z'8 9'8!o'8 9HHj/:$GM1/Ba% 1,?II,, IIlkAo6 7RW-   !Q &-q0N.q1NNN (;(MUWU]U] ^ao o p  !!"34/&2"451a:&+(V,33JVa@abii $T4]3Goog 'UV^^+5G,g5/A"55GVYZGZ!0CCDmB? w:s( I0features_shape num_negativesmask_time_indicescLUup4[R"U5n[R"X4U4[RS9nUbUR [ 5O[R "U[ S9n[U5HnX'R5S- nXRUn [R"[R"US-5SS2S4US-U45n [RRSXS-U4S9n XU :==S- ss'XXgX''Xg==Xt-- ss'M U$)z6 Sample `num_negatives` vectors from feature vectors. )r=rPNrOr!r)size) rSr^r[rbastyper\rarZrXrerTrandint) rrrsrtrhrKsequence_length_rangesampled_negative_indices batch_idxhighmapped_masked_indicesfeature_indicessampled_indicess r<_sample_negative_indicesrs5#1JIIo6 "xxzM.Zbdbjbjk+<*G  &RWWUckoMp:&  +//1A5 5 6R S//"))D1H*=ag*FPQS`Hab))++At!8]:S+T?:;q@;MbLr +,=,HI !+y/JJ+' $#r;c2^\rSrSrSU4SjjrSrSrU=r$)Wav2Vec2NoLayerNormConvLayercb>[TU]5 US:aURUS- OSUlURUUl[ R "URURURUURUURS9Ul [URUl g)Nrr! kernel_sizestridebias)super__init__conv_dim in_conv_dim out_conv_dimrConv1d conv_kernel conv_stride conv_biasconvr feat_extract_activation activationselfconfiglayer_id __class__s r<r%Wav2Vec2NoLayerNormConvLayer.__init__s [TU]5 US:aURUS- OSUlURUUl[ R "URURURUURUURS9Ul [ R"URSS9Ul [URUlg)Nrr!rT)elementwise_affine)rrrrrrrrrrr LayerNorm layer_normr rrrs r<r#Wav2Vec2LayerNormConvLayer.__init__s [TU]5 US:aURUS- OSUlURUUl[ R "URURURUURUURS9Ul [URUl [ R"URURSS9Ulg)Nrr!rT) num_groups num_channelsaffine)rrrrrrrrrrrr rr GroupNormrrs r<r#Wav2Vec2GroupNormConvLayer.__init__/s [TU]5 [R"URURUR UR S-UR S9Ul[RRn[[RRS5(a$[RRRn[5(Ga%SSK nURRURR SS9 U"URSSS9UlSSS5 [URS5(aUURRR R"nURRR R$nO,URR&nURR(nURR+X5 URR+X5 OU"URSSS9Ul[-UR 5Ul[0UR2Ulg!,(df  GN,=f) Nr#)rpaddinggroups weight_normr) modifier_rankweight)namedimparametrizations)rrrr hidden_sizenum_conv_pos_embeddingsnum_conv_pos_embedding_groupsrutilsrhasattrrr deepspeedzeroGatheredParametersr original0 original1weight_gweight_vregister_external_parameterWav2Vec2SamePadLayerrr rr)rrrrrrrs r<r(Wav2Vec2PositionalConvEmbedding.__init__Gs II      6622a777   hh** 288,,m < <((33??K % ' ' 224993C3CST2U' aH Vtyy"4559955<<FF9955<<FF99--99-- NN 6 6t F NN 6 6t F#DIIH!DDI+F,J,JK  !?!?@VUs I IcURSS5nURU5nURU5nURU5nURSS5nU$Nr!r#)rrrrrs r<r'Wav2Vec2PositionalConvEmbedding.forwardhsV%//15  -0  ]3  6 %//15 r;)rrrrrs@r<rrFsABr;rc.^\rSrSrU4SjrSrSrU=r$)riscR>[TU]5 US-S:XaSUlgSUlg)Nr#rr!)rrnum_pad_remove)rrrs r<rWav2Vec2SamePadLayer.__init__ts) #:Q#>!#Car;cXURS:aUSS2SS2SUR*24nU$Nrrrs r<rWav2Vec2SamePadLayer.forwardxs6    ")!Q0F43F3F2F0F*FGMr;rrrs@r<rrssKr;rc8^\rSrSrSrU4SjrSrSrSrU=r $)Wav2Vec2FeatureEncoderi~z.Construct the features from raw audio waveformc >[TU]5 URS:Xa@[USS9/[ UR S- 5Vs/sHn[ XS-S9PM sn-nOVURS:Xa-[ UR 5Vs/sH n[XS9PM nnO[SURS35e[R"U5Ul SUl S Ul gs snfs snf) Ngroupr)rr!layerz`config.feat_extract_norm` is z), but has to be one of ['group', 'layer']FT)rrfeat_extract_normrrZnum_feat_extract_layersrrrRr ModuleList conv_layersgradient_checkpointing_requires_grad)rrirrs r<rWav2Vec2FeatureEncoder.__init__s   # #w .5fqIJNSTZTrTruvTvNwNNw,V!eDNwNK % % 0HMfNlNlHmHm1*6>Hm K01I1I0JJst ==5&+#"Ns C C%cNUR5H nSUlM SUlgNF) parameters requires_gradrrparams r<_freeze_parameters)Wav2Vec2FeatureEncoder._freeze_parameterss#__&E"'E '#r;cUSS2S4nUR(aUR(aSUlURH nU"U5nM U$)NT)rtrainingrr)r input_valuesr, conv_layers r<rWav2Vec2FeatureEncoder.forwardsK$QW-    4==*.M '**J&}5M+r;)rrr) r1r2r3r4r5rrrr:rrs@r<rr~s8#&$  r;rc.^\rSrSrU4SjrSrSrU=r$)Wav2Vec2FeatureProjectionic4>[TU]5 [R"URSUR S9Ul[R"URSUR5Ul [R"UR5Ul g)NrNeps) rrrrrlayer_norm_epsrLinearr projectionDropoutfeat_proj_dropoutdropoutrrrs r<r"Wav2Vec2FeatureProjection.__init__sf ,,vr':@U@UV))FOOB$79K9KLzz&":":; r;cnURU5nURU5nURU5nX4$r)rrr)rr,norm_hidden_statess r<r!Wav2Vec2FeatureProjection.forwards7!__];(:;  ]3 00r;)rrrrrs@r<rrs< 11r;rmodulequerykeyvaluescalingrkwargsc UcURS5S-n[R"XRSS55U-nUbX-n[R R USS9n[R RXURS9n[R"X5n U RSS5R5n X4$)NrNr#rr)prr!) rvr6matmulrr functionalsoftmaxrr contiguous) r r r r r@r rr attn_weights attn_outputs r<eager_attention_forwardrs**R.D(<<}}Q':;gEL!#4 ==((2(>L==((6??([L,,|3K''1-88:K  $$r;c:^\rSrSrSrSS\S\S\S\S\S \S \S-4U4S jjjr SS \ RS \ RS-S\ RS-S\S-S\ \ S\\ R\ RS-\\ RS-44 SjjrSrU=r$)Wav2Vec2Attentioniz=Multi-headed attention from 'Attention Is All You Need' paperN embed_dim num_headsr is_decoderr is_causalrc>[TU]5 XlX lX0lX-UlXplUR U-UR:wa[SURSUS35eUR S-UlX@l X`l [R"XUS9Ul [R"XUS9Ul[R"XUS9Ul[R"XUS9Ulg)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r)r)rrrrrhead_dimrrRr rrrrk_projv_projq_projout_proj) rrrrrrrrrs r<rWav2Vec2Attention.__init__s "" !.  MMI %$.. 8MdnnM]$YKr3 }}d* $"ii 4@ ii 4@ ii 4@  )TB r;r,key_value_statesr@output_attentionsrrBc USLnURSSn/UQSPURP7nURU5RU5R SS5n U(aUOUn /U RSSQSPURP7n UR U 5RU 5R SS5n UR U 5RU 5R SS5n [R"URR[5nU"UU U U U4UR(dSO URURUS.UD6unnUR"/UQSP76R!5nUR#U5nUUS4$)z#Input shape: Batch x Time x ChannelNrNr!r#)rr r()r=r!r$viewrr"r#r get_interfacer_attn_implementationrrrr rfrr%)rr,r'r@r(ris_cross_attention input_shape hidden_shape query_statescurrent_stateskv_shape key_states value_statesattention_interfacerrs r<rWav2Vec2Attention.forwards.T9$))#2. 88b8$--8 {{=166|DNNqRST -?)]B^))#2.BBDMMB[[055h?II!QO {{>277AKKAqQ (?(M(M KK , ,.E) %8      % $}}C$,,LL/ %  % ! \"));;;;FFH mmK0 L$..r;) rrrr!rrr"rr%r$r r#)r*FTFN)NNF)r1r2r3r4r5rFfloatr\r"rr6Tensorrrr9rr:rrs@r<rrs G  (,CCC C  C  CC%CCD15.2). 0/||0/ ,,-0/ t+ 0/  $; 0/-.0/ u||U\\D0% 2E2LL M0/0/r;rc.^\rSrSrU4SjrSrSrU=r$)Wav2Vec2FeedForwardi'c>[TU]5 [R"UR5Ul[R "URUR5Ul [UR[5(a[URUlOURUl[R "URUR5Ul[R"UR 5Ulgr)rrrractivation_dropoutintermediate_dropoutrrintermediate_sizeintermediate_dense isinstance hidden_actstrr intermediate_act_fn output_densehidden_dropoutoutput_dropoutrs r<rWav2Vec2FeedForward.__init__(s $&JJv/H/H$I!"$))F,>,>@X@X"Y f'' - -'-f.?.?'@D $'-'8'8D $IIf&>&>@R@RS jj)>)>?r;cURU5nURU5nURU5nURU5nUR U5nU$r)r@rDr>rErGrs r<rWav2Vec2FeedForward.forward5sX// > 00? 11-@ ))-8 ++M: r;)rDr@r>rErGrrs@r<r;r;'s @r;r;c2^\rSrSrU4SjrSSjrSrU=r$)Wav2Vec2EncoderLayeri?c>[TU]5 [URURUR SUS9Ul[R"UR5Ul [R"URURS9Ul [U5Ul[R"URURS9Ulg)NFrrrrrr)rrrrnum_attention_headsattention_dropout attentionrrrFrrrrr; feed_forwardfinal_layer_normrs r<rWav2Vec2EncoderLayer.__init__@s *((00,,  zz&"7"78 ,,v'9'9v?T?TU/7 " V-?-?VEZEZ [r;cUnURXUS9upnURU5nXA-nURU5nXRU5-nUR U5nU4nU(aXu4- nU$Nr@r()rQrrrRrSrr,r@r( attn_residualrrioutputss r<rWav2Vec2EncoderLayer.forwardOs% )- L]*8* & Q ]3 %5  6 %(9(9-(HH --m<  "   &Gr;)rQrrRrSrrrrs@r<rLrL?s \r;rLct^\rSrSrU4SjrS S\R S\R S-S\4SjjrSr U=r $) #Wav2Vec2EncoderLayerStableLayerNormicc>[TU]5 [URURUR SUS9Ul[R"UR5Ul [R"URURS9Ul [U5Ul[R"URURS9Ul[#USS5b[%U5UlgSUlg)NFrNradapter_attn_dim)rrrrrOrPrQrrrFrrrrr;rRrSgetattrWav2Vec2AttnAdapterLayer adapter_layerrs r<r,Wav2Vec2EncoderLayerStableLayerNorm.__init__ds *((00,,  zz&"7"78 ,,v'9'9v?T?TU/7 " V-?-?VEZEZ [ 6-t 4 @!9&!AD !%D r;Nr,r@r(cUnURU5nURXUS9upnURU5nXA-nXRUR U55-nUR bXR U5-nU4nU(aXu4- nU$rV)rrQrrRrSrbrXs r<r+Wav2Vec2EncoderLayerStableLayerNorm.forwardws &  6 )- L]*8* & Q ]3 %5 %(9(9$:O:OP]:^(__    )),>,>},MMM "   &Gr;)rbrQrrRrSrr) r1r2r3r4rr6r9r\rr:rrs@r<r]r]csC&,/3"' || t+ r;r]c ^\rSrSrU4SjrS S\R S\RS-S\S\S\4 S jjr S r U=r $) Wav2Vec2Encoderic>[TU]5 Xl[U5Ul[ R "URURS9Ul [ R"UR5Ul [ R"[UR5Vs/sHn[!U5PM sn5UlSUlgs snfNrF)rrrrpos_conv_embedrrrrrrrFrrrZnum_hidden_layersrLlayersrrrrirs r<rWav2Vec2Encoder.__init__s  =fE,,v'9'9v?T?TUzz&"7"78 mm5QWQiQiKj$kKja%9&%AKj$kl &+#%l C Nr,r@r(output_hidden_states return_dictc,U(aSOSnU(aSOSnUb4URS5RSSURS5nSX)'[URUUS9nUR U5n XR UR5-nURU5nURU5n[5=(d [U5n URHn U(aXa4-n[R"/5n UR=(a XRR :n U (aU (a U "XUS9nUSnU (aSnU(dM|UWS4-nM U(aXa4-nU(d[#S XU455$[%UUUS 9$) Nr0rNr!r#rr inputs_embedsr@rWNNc3.# UH ocMUv M g7frr0.0vs r< *Wav2Vec2Encoder.forward..m$[q$[ last_hidden_stater,r-) unsqueezerepeatr=r rrjtodevicerrr r rlr6rUr layerdropr9rrr,r@r(rprqall_hidden_statesall_self_attentionsexpand_attention_maskposition_embeddings synced_gpusrdropout_probabilityskip_the_layer layer_outputss r<rWav2Vec2Encoder.forwards#7BD$5b4  %$2$<$(>}?S?S(TT  6  ]3 02R6LT6R [[E#$58H$H!#(**R. !]]Z/B[[EZEZ/ZN![ %!Te! !.a 0  ,   &9]1=M[TU]5 Xl[U5Ul[ R "URURS9Ul [ R"UR5Ul [ R"[UR5Vs/sHn[!U5PM sn5UlSUlgs snfri)rrrrrjrrrrrrrFrrrZrkr]rlrrms r<r'Wav2Vec2EncoderStableLayerNorm.__init__s  =fE,,v'9'9v?T?TUzz&"7"78 mmBGH`H`Ba bBaQ 0 8Ba b  ',# crocU(aSOSnU(aSOSnUb4URS5RSSURS5nSX)'[URUUS9nUR U5n X-nUR U5n[5=(d [U5n URHn U(aXa4-n[R"/5n UR=(a XRR:n U (aU (a U "XUS9nUSnU (aSnU(dM|UWS4-nM URU5nU(aXa4-nU(d[S XU455$[!UUUS 9$) Nr0rNr!r#rrsrWruc3.# UH ocMUv M g7frr0rws r<rz9Wav2Vec2EncoderStableLayerNorm.forward..r|r}r~)rrr=r rrjrr r rlr6rUrrrr9rrs r<r&Wav2Vec2EncoderStableLayerNorm.forwards#7BD$5b4  %$2$<$[TU]5 URUlURUlUR UR-S:wa&[SUR SURS35e[R"[R"SURUR -UR UR-55Ul [R"URSURUR -5UlSUlg)Nrz`config.codevector_dim z5 must be divisible by `config.num_codevector_groups` z for concatenationr!rNr#)rrnum_codevector_groupsrnum_codevectors_per_groupnum_varscodevector_dimrRr Parameterr6r7 codevectorsrr weight_proj temperaturers r<r&Wav2Vec2GumbelVectorQuantizer.__init__+s  6688  4?? 2a 7)&*?*?)@A559__4EEWY  <<   a4==!@&BWBW[_[j[jBj k 99V__R%8$//DMM:YZr;cUb}UR5SS2SS4RUR5n[R"X [R "U55nUR SS9UR 5- nOURSS9n[R"[R "[R"X35SS9*5R 5nU$)NrrrN) flattenexpandr=r6where zeros_likerXmeanexpxlogy)probsmask mask_extendedmarginal_probs perplexitys r<_compute_perplexity1Wav2Vec2GumbelVectorQuantizer._compute_perplexity?s   LLN1dD=9@@MMKK e6F6Fu6MNE"YY1Y- :N"ZZAZ.NYY %++n*U[] ^^_cce r;cURup4nURU5nURX4-UR-S5nUR(a[ R RUR5URSS9RU5n[R"URX4-URS5R5SS9nURXr5nO{URSS9n URUR5R!SU RSS5S5nURX4-URS5nURXb5nURX4-S5nUR#S5UR$-n U RX4-URUR&S5n U R)S5RX4S5n X4$)NrNT)tauhardrr!g?r)r=rr+rrrrgumbel_softmaxr8rtype_asr6rrargmax new_zerosscatter_rrrrX) rr,rtrhrKrcodevector_probscodevector_soft_distrcodevector_idxcodevectors_per_grouprs r<r%Wav2Vec2GumbelVectorQuantizer.forwardKs3@3F3F0 [((7 %**:+G$//+Y[]^ ==!}};;##%4+;+;$ < gm$  $)=="":#?RTU[[]ce$ 112FZJ+11b19N,66}7J7JKTTN''A.   044Z5QSWSbSbdfg 112BVJ+001MrR 0 : :2 >AQAQ Q+001Mt`d`m`moqr !oob)..zBO &&r;)rrrrrr) r1r2r3r4r5r staticmethodrrr:rrs@r<rr%s+ (  #'#'r;rc.^\rSrSrU4SjrSrSrU=r$)Wav2Vec2Adapteriqc>^[TU]5 TRTR:waV[R "TRTR5Ul[R"TR5UlOS=UlUl[R"U4Sj[TR555Ul TRUl g)Nc3:># UHn[T5v M g7fr)Wav2Vec2AdapterLayer)rxrirs r<rz+Wav2Vec2Adapter.__init__..|s#kJjQ$8$@$@Jjs)rroutput_hidden_sizerrrprojrproj_layer_normrrZnum_adapter_layersrlrrs `r<rWav2Vec2Adapter.__init__rs   $ $(:(: : &"4"4f6O6OPDI#%<<0I0I#JD /3 3DI,mm#k%PVPiPiJj#kk ))r;c|URb/URb"URU5nURU5nURSS5nURHKn[R R 5nUR (aX0R:dMCU"U5nMM URSS5nU$r)rrrrlrSrTrr)rr,rlayerdrop_probs r<rWav2Vec2Adapter.forwards 99 T%9%9%E IIm4M 00?M%//15 [[EYY--/N==^nn%D %m 4 ! &//15 r;)rrlrrrrs@r<rrqs *r;rc.^\rSrSrU4SjrSrSrU=r$)ric>[TU]5 [R"URSUR-UR UR SS9Ulg)Nr#r!)rr)rrrrradapter_kernel_sizeadapter_striderrs r<rWav2Vec2AdapterLayer.__init__sJ II  % % )) )  & &((   r;cdURU5n[RRUSS9nU$)Nr!r)rrrglurs r<rWav2Vec2AdapterLayer.forwards/ -0  ))-Q)? r;)rrrs@r<rrs r;rcJ^\rSrSrU4SjrS\R 4SjrSrU=r $)raic>[TU]5 URUlURUl[ R"UR 5Ul[ R"UR UR5Ul [ R"5Ul [ R"URUR 5Ul g)z Implements adapter modules directly with 3D tensor weight as parameters and without using ModuleList to speed up training throughput. N)rrr_ input_dimr hidden_dimrrnormrlinear_1ReLUact_fnlinear_2rs r<r!Wav2Vec2AttnAdapterLayer.__init__s 00 ,,LL1  $//4>>B ggi  $..$//B r;r,cURU5nURU5nURU5nURU5nU$r)rrrrrs r<r Wav2Vec2AttnAdapterLayer.forwards@ -0  m4  M2  m4 r;)rrrrrr) r1r2r3r4rr6r7rr:rrs@r<raras  CU%6%6r;rac\rSrSr%\\S'SrSrSrSr Sr Sr Sr \ R"5S5rSS \ R \-S \S-4S jjrSS \S \ R 4SjjrSrSrSS\4SjjrSrg)Wav2Vec2PreTrainedModelirwav2vec2raudioTc [U[5(a5URR5 URR5 g[U[ 5(au[ R"URRSSS9 [ R"URR5 [ R"UR5 g[U[5(a[ R"URRSS[ R""SURR$SURR&-- 5-S9 [ R("URRS5 g[U[*5(a[ R""SUR,R.- 5n[ R"UR,RU*US9 [ R"UR,RU*US9 g[U[0R25(ac[ R"URSUR4R6S9 URb![ R"UR5 gg[U[0R8[0R:45(aA[ R"UR5 [ R<"UR5 g[U[0R>5(a[ R@"UR5 URb_[ R""URBUR&UR$S-- 5n[ R"URU*US9 ggg)zInitialize the weightsr*r!)rstdrr#)abN)"rAWav2Vec2ForPreTraining project_hidreset_parameters project_qrinitnormal_rrzeros_runiform_rrrmathsqrtr in_channels constant_rr in_featuresrrrinitializer_rangerrones_rkaiming_normal_r)rr ks r< _init_weights%Wav2Vec2PreTrainedModel._init_weightssn f4 5 5    / / 1    - - /  = > > LL++22! D KK**// 0 MM&,, -  ? @ @ LL "" !v{{'>'>q'AFKKD[D['["\]]  NN6;;++Q /  9 : : !f//;;; MM&++00QB! <  * * LLSdkk6S6S T{{& FKK(' r|| < = = KK $ JJv}} %  * *   /{{&IIfmmv/A/AFDVDVWXDY/YZ[ fkkaR15'+r;Nrj add_adaptercdUcURROUnSn[URRURR5HupEU"XU5nM U(aD[ URR 5H!nU"USURR5nM# U$)z8 Computes the output length of the convolutional layers c8[R"X- USS9S-$)Nfloor) rounding_moder!)r6divrHrrs r<_conv_out_lengthRWav2Vec2PreTrainedModel._get_feat_extract_output_lengths.._conv_out_lengths 99\7wWZ[[ [r;r!)rrziprrrZrr)rrjrrrrris r< _get_feat_extract_output_lengths8Wav2Vec2PreTrainedModel._get_feat_extract_output_lengthss 2=1Ddkk--+  \ $'t{{'>'> @W@W#X K,]PM$Y 4;;99: 04;;C]C] ^ ;r;feature_vector_lengthr@cURSS9SS2S4nURXCS9nUR[R5nUR Sn[R "Xa4URURS9nSU[R"UR SURS9US- 4'URS/5RS5RS/5R5nU$)NrNrrr)rPrr!)r) cumsumr rr6longr=r[rPrr^flipr\)rr r@rnon_padded_lengthsoutput_lengthsrhs r<"_get_feature_vector_attention_mask:Wav2Vec2PreTrainedModel._get_feature_vector_attention_masks ,22r2:1b5A>>?Q>k'**5::6#))!,   /~7K7KTbTiTi uv ^%9%9!%<^EZEZ[]kno]opq',,bT299"=BBB4HMMOr;cURRc[URS35e0nUR 5HIup#[ U[ 5(dMUR5HupEXQSRX$/5'M MK [ U[5(a8URR5Hup%XQSRSU/5'M U$)NzF has no adapter layers. Make sure to define `config.adapter_attn_dim`..lm_head) rr_rRr named_modulesrAranamed_parametersjoinWav2Vec2ForCTCr)radapter_weightsrr  param_namers r< _get_adapters%Wav2Vec2PreTrainedModel._get_adapterss ;; ' ' ///uvw w ..0LD&":;;)/)@)@)B%JDICHHd-?$@A*C1 dN + +#||<<> ?D)T): ;< ?r;cUR5H+n[U[5(dMURU5 M- [U[5(aURUR 5 gg)zS (Re-)initialize attention adapter layers and lm head for adapter-only fine-tuning N)modulesrArarrr)rr s r<init_adapter_layers+Wav2Vec2PreTrainedModel.init_adapter_layers%sV llnF&":;;""6*% dN + +   t|| , ,r; target_langc URRc[SUS35eXR:Xa!U(d[R SUS35 gUR SS5nUR SS5nUR S S5nUR S S5nUR S S5nUR S S5n UR S S5n URRn Sn U SLa1[RU5n [U U UUUUU US9n[U5n U cF[RU5n [U U UUUUU US9n[5 [ R""USSS9n UR%5n['U R)55['UR)55- n['UR)55['U R)55- n[+U5S:a![SWSSR-U5S35e[+U5S:a![SWSSR-U5S35eU SR.SnUURR0:waU[2R4"URR6UUR8UR:S9UlUURlU R?5VVs0sHunnUURAUU5_M n nnURCU SS9 Xlg![a U (aeGN[a U (a[SU SU SU S35eGN,f=f![a e[a e[a [SU SU SU S35ef=fs snnf)a Load a language adapter model from a pre-trained adapter model. Parameters: target_lang (`str`): Has to be a language id of an existing adapter weight. Adapter weights are stored in the format adapter..safetensors or adapter..bin force_load (`bool`, defaults to `True`): Whether the weights shall be loaded even if `target_lang` matches `self.target_lang`. cache_dir (`Union[str, os.PathLike]`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist. proxies (`dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. local_files_only(`bool`, *optional*, defaults to `False`): Whether or not to only look at local files (i.e., do not try to download the model). token (`str` or `bool`, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, or not specified, will use the token generated when running `hf auth login` (stored in `~/.huggingface`). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. To test a pull request you made on the Hub, you can pass `revision="refs/pr/"`. mirror (`str`, *optional*): Mirror source to accelerate downloads in China. If you are from China and have an accessibility problem, you can set this option to resolve it. Note that we do not guarantee the timeliness or safety. Please refer to the mirror site for more information. Activate the special ["offline-mode"](https://huggingface.co/transformers/installation.html#offline-mode) to use this method in a firewalled environment. Examples: ```python >>> from transformers import Wav2Vec2ForCTC, AutoProcessor >>> ckpt = "facebook/mms-1b-all" >>> processor = AutoProcessor.from_pretrained(ckpt) >>> model = Wav2Vec2ForCTC.from_pretrained(ckpt, target_lang="eng") >>> # set specific language >>> processor.tokenizer.set_target_lang("spa") >>> model.load_adapter("spa") ``` NzCannot load_adapter for - if `config.adapter_attn_dim` is not defined.z#Adapter weights are already set to r cache_dirforce_downloadFproxieslocal_files_onlytokenrevisionuse_safetensors)filenamer)r*r+r,r-r(zCan't load the model for 'z'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'z=' is the correct path to a directory containing a file named cpuT) map_location weights_onlyrzThe adapter weights z has unexpected keys: z, z has missing keys: zlm_head.weightrrP)strict)"rr_rRr%loggerwarningpop _name_or_pathWAV2VEC2_ADAPTER_SAFE_FILEformatrsafe_load_fileOSError ExceptionWAV2VEC2_ADAPTER_PT_FILErr6loadrsetkeysr_rr= vocab_sizerrrrrPritemsrload_state_dict)rr% force_loadrr(r)r*r+r,r-r.model_path_or_id state_dictfilepath weight_pathrunexpected_keys missing_keystarget_vocab_sizerrys r< load_adapter$Wav2Vec2PreTrainedModel.load_adapter2sx ;; ' ' /7 }Dqrs s ** *: NN@ QO P JJ{D1 $4e<**Y-!::&8%@ 7D)::j$/ **%6=;;44  % '188EH )$%#1#%5%'  ,K8 &  /66{CH" )$%#1#%5%'  )*"ZZ!&!% .,,.joo/037K7K7M3NN?//12S9J5KK  ! #3K=@VW[W`W`apWqVrrstu u   "3K=@STXT]T]^jTkSllmno o''78>>qA  6 6 699 ..0A$++]a]g]gDL&7DKK "?I>N>N>PQ>Pdaaoa011>P Q Z6'] "#  "!45E4FG==M>FZqJ# B   01A0BC99I8JK::B1F 6Rs*L<0M"N M'%MM4N)rr%r)T)r1r2r3r4r"r8base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attnr6no_gradr LongTensorrFr\r rrr#rCrMr:r0r;r<rrs "$O&*#N ]]_#6#6Je>N>NQT>Tcgjncn,Y]%(:?:J:J( -m'm'm'r;rc@^\rSrSrS\4U4SjjrSrSS\RS\RS-S\RS-4S jjr \ SS \RS-S\RS-S\RS-S \ S-S \ S-S \ S-S\\-4Sjj5rSrU=r$) Wav2Vec2Modelirc>[TU]U5 Xl[U5Ul[ U5UlURS:dURS:aG[R"[R"UR5R55UlUR (a[#U5UlO['U5UlUR((a [+U5OSUlUR/5 gNr*)rrrrfeature_extractorrfeature_projectionmask_time_probmask_feature_probrrr6r9rrmasked_spec_embeddo_stable_layer_normrencoderrgrradapter post_initrs r<rWav2Vec2Model.__init__s   !7!?";F"C  3 &&*B*BS*H%'\\%,,v?Q?Q2R2[2[2]%^D "  & &9&ADL*62DL282D2Dv.$  r;c8URR5 gz Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training. N)r\rrs r<freeze_feature_encoder$Wav2Vec2Model.freeze_feature_encoders 113r;Nr,rtr@c[URSS5(dU$UR5upEnUb(URR UR 5X'OURR S:aUR(a[XE4URR URRUURRS9n[R"X!R[RS9nURR UR 5X'URRS:aUR(a[XF4URRURR URR"S9n[R"XqR[RS9nUSS2S4R%SUS5nSX'U$) z Masks extracted features along time axis and/or along feature axis according to [SpecAugment](https://huggingface.co/papers/1904.08779). apply_spec_augmentTNr)r>r?r@rAr3)r>r?rArN)r`rrvr`rrPr^rrqmask_time_lengthmask_time_min_masksr6rrr\r_mask_feature_lengthmask_feature_min_masksr)rr,rtr@rhrKrmask_feature_indicess r<_mask_hidden_states!Wav2Vec2Model._mask_hidden_statesst{{$8$?? 4A3E3E3G0 [  (/3/E/E/H/HI\I\/]M , [[ ' '! + 5-++44 KK88-++99 ! !& ->G[G[chcmcm n /3/E/E/H/HI\I\/]M , ;; ( (1 ,#8)++77 KK;;++<< $ $)<<0DMaMainisis#t #74#@#G#GO]_#` 23M /r;rr(rprqrBc >UbUOURRnUbUOURRnUbUOURRnUR U5nUR SS5nUbUR URSUSS9nURU5upURXUS9n URU UUUUS9n U Sn URbURU 5n U(d X4U SS-$[U UU RU RS 9$) a mask_time_indices (`torch.BoolTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices to mask extracted features for contrastive loss. When in training mode, model learns to predict masked extracted features in *config.proj_codevector_dim* space. Nr!r#Fr)rtr@r@r(rprqr)rextract_featuresr,r-)rr(rprqr\rrr=r]rrrbrcrr,r-) rrr@rtr(rprqrrvr,encoder_outputss r<rWav2Vec2Model.forward-sY 2C1N-TXT_T_TqTq$8$D $++JjJj &1%uEN+/*A*ABR*S' 00 ~1 ,, )/!5# ' (* << # LL7M!4qr7JJ J&+-)77&11   r;)rcrrbr\r]r`ruNNNNN)r1r2r3r4r"rrir6r7rWrrrr9r\r9rrr:rrs@r<rYrYs~(47;26 ,((,!,,t3,((4/ ,\/36:)-,0#'8 llT)8  t+8 !,,t3 8  $; 8 #Tk 8 D[8  ( (8 8 r;rYz? Wav2Vec2 Model with a quantizer and `VQ` head on top. cn^\rSrSrS\4U4SjjrS\4SjrSr\ SS\ RS\ RS \ RS\ 4S jj5r \SS \ RS -S \ RS -S\ R S -S\ R S -S\S -S\S -S\S -S\\-4Sjj5rSrU=r$)riirc>[TU]U5 [U5Ul[R "UR 5Ul[U5Ul [R"URUR5Ul [R"URUR5UlUR!5 gr)rrrYrrrfeat_quantizer_dropoutdropout_featuresr quantizerrrproj_codevector_dimrrrrdrs r<rWav2Vec2ForPreTraining.__init__os  %f- " 6+H+H I6v>99V%7%79S9ST6#8#8&:T:TU r;rc$XRlg)zR Set the Gumbel softmax temperature to a given value. Only necessary for training N)r~r)rrs r<set_gumbel_temperature-Wav2Vec2ForPreTraining.set_gumbel_temperature|s&1"r;cLURRR5 grgrr\rrhs r<ri-Wav2Vec2ForPreTraining.freeze_feature_encoder ''::>> import torch >>> from transformers import AutoFeatureExtractor, Wav2Vec2ForPreTraining >>> from transformers.models.wav2vec2.modeling_wav2vec2 import _compute_mask_indices, _sample_negative_indices >>> from datasets import load_dataset >>> feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base") >>> model = Wav2Vec2ForPreTraining.from_pretrained("facebook/wav2vec2-base") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> input_values = feature_extractor(ds[0]["audio"]["array"], return_tensors="pt").input_values # Batch size 1 >>> # compute masked indices >>> batch_size, raw_sequence_length = input_values.shape >>> sequence_length = model._get_feat_extract_output_lengths(raw_sequence_length).item() >>> mask_time_indices = _compute_mask_indices( ... shape=(batch_size, sequence_length), mask_prob=0.2, mask_length=2 ... ) >>> sampled_negative_indices = _sample_negative_indices( ... features_shape=(batch_size, sequence_length), ... num_negatives=model.config.num_negatives, ... mask_time_indices=mask_time_indices, ... ) >>> mask_time_indices = torch.tensor(data=mask_time_indices, device=input_values.device, dtype=torch.long) >>> sampled_negative_indices = torch.tensor( ... data=sampled_negative_indices, device=input_values.device, dtype=torch.long ... ) >>> with torch.no_grad(): ... outputs = model(input_values, mask_time_indices=mask_time_indices) >>> # compute cosine similarity between predicted (=projected_states) and target (=projected_quantized_states) >>> cosine_sim = torch.cosine_similarity(outputs.projected_states, outputs.projected_quantized_states, dim=-1) >>> # show that cosine similarity is much higher than random >>> cosine_sim[mask_time_indices.to(torch.bool)].mean() > 0.5 tensor(True) >>> # for contrastive loss training model should be put into train mode >>> model = model.train() >>> loss = model( ... input_values, mask_time_indices=mask_time_indices, sampled_negative_indices=sampled_negative_indices ... ).loss ```N)r@r(rprtrqrr!Fr)rtrNr#rz-infirX) reduction)r(r)r*r+r,r-r.r/)$rrqrr6r\rrr}rr=r~rrrPr+rpermutercontrastive_logits_temperatureallanyr8rrfrvrrr cross_entropyrrrXdiversity_loss_weightr&r,r-)rrr@rtrzr(rprqrrZtransformer_featuresrvquantized_featuresr+r(r.r/rhrKrnegative_quantized_featuresr neg_is_postargetnum_codevectorss r<rWav2Vec2ForPreTraining.forwards;F&1%uEN59NN 5C5 10224>>3H3H3N3NO!^^,>?3777. # /7I7O7O 4J +=*A*A"k*R(--/44R8+ '+F*J*JOR+gaAq! ( 44"47++$ :: F-0KKPPQSTJ~~).vqr :&%%a+33B AGF,1133t;FFq!LTTVF!}}::6<<>6]b:c "kkCCdkkFgFggO.1FF/Y]n]r]r]ttN$kk&G&G.&XXD4F^ahijikalll(>STW^_`_aWbb b+1'9"7!//))-)  r;)r}rrr~r)g?)NNNNNN)r1r2r3r4r"rrFrrirr6r7r8rrr9 BoolTensorr\r9r&rr:rrs@r<rris) ~ 1#1 = ! ** ,,"-- (/359<@)-,0#'] llT)]  t+] !++d2 ] #("2"2T"9 ]  $; ] #Tk] D[]  - -] ] r;rzp Wav2Vec2 Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC). c^\rSrSrSS\S-4U4SjjjrSrSrSr\ SS\ RS-S \ RS-S \ S-S \ S-S \ S-S \ RS-S\ \-4Sjj5rSrU=r$)ri?Nr%c>[TU]U5 [U5Ul[R "UR 5UlX lURc[SURS35e[US5(aUR(a URO URn[R "X1R5UlUR%5 g)a  target_lang (`str`, *optional*): Language id of adapter weights. Adapter weights are stored in the format adapter..safetensors or adapter..bin. Only relevant when using an instance of [`Wav2Vec2ForCTC`] with adapters. Uses 'eng' by default. NzYou are trying to instantiate z with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `Wav2Vec2ForCTC.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration.r)rrrYrrr final_dropoutrr%rBrRrrrrrrrrd)rrr%rrs r<rWav2Vec2ForCTC.__init__Es  %f- zz&"6"67 &    $00@AHH *1)G)GFL^L^F % %djdvdv yy!35F5FG  r;c @[5[R"S5:XagURnUb'[ UR SS5c[ SUS35eUc.[ UR SS5b[RS5 gUbURUSS9 gg) a This method overwrites [`~PreTrainedModel.tie_weights`] so that adapter weights can be correctly loaded when passing `target_lang=...` to `from_pretrained(...)`. This method is **not** supposed to be called by the user and is prone to be changed in the future. metaNr_zCannot pass `target_lang`: r'z)By default `target_lang` is set to 'eng'.T)rE) rr6rr%r`rrRr5inforM)rrr%s r< tie_weightsWav2Vec2ForCTC.tie_weightsbs 6 75<<;O O  &&  "wt{{[TU]U5 [US5(aUR(a [ S5e[ U5UlURS-nUR(a2[R"[R"U5U- 5Ul [R"URUR 5Ul[R"UR UR$5UlUR)5 g)Nrz_Sequence classification does not support the use of Wav2Vec2 adapters (config.add_adapter=True)r!)rrrrrRrYrrkuse_weighted_layer_sumrrr6ra layer_weightsrrclassifier_proj_size projector num_labels classifierrdrr num_layersrs r<r*Wav2Vec2ForSequenceClassification.__init__s  6= ) )f.@.@q &f- --1  ( (!#ejj.Dz.Q!RD 6#5#5v7R7RS))F$?$?ARARS r;cLURRR5 grgrrhs r<ri8Wav2Vec2ForSequenceClassification.freeze_feature_encoderrr;cTURR5H nSUlM grrrs r<r3Wav2Vec2ForSequenceClassification.freeze_base_modelrr;Nrr@r(rprqrrBc 0UbUOURRnURR(aSOUnURUUUUUS9nURR(aiU[n [ R "U SS9n [RRURSS9n XRSSS5-RSS9n OUSn URU 5n UcU RSS9n OURU R SU5n U R#S5R%SSU R S5n S X)'U RSS9U RSS9RSS5- n UR'U 5nSnUbF[)5nU"URSURR*5URS55nU(dU4U[S-nUbU4U-$U$[-UUUR.UR0S 9$)  input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoProcessor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`Wav2Vec2Processor.__call__`] for details. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NTrur!rrNrr#r*r)rrqrrrr6stackrrrrr+rXrrrr=rrrrrrr,r-)rrr@r(rprqrrrZr, norm_weights pooled_output padding_maskexpand_padding_maskrr(loss_fctrs r<r)Wav2Vec2ForSequenceClassification.forwards0&1%->r1a-HHMMRSMTM#AJM}5  !)..1.5MBB=CVCVWXCY[ijL"."8"8"<"C"CAq-J]J]^_J`"a 25M. /)--!-4|7G7GA7G7N7S7STVXY7ZZM/  ')HFKKDKK,B,BCV[[QS_UDY)F)G!HHF)-)9TGf$ Ev E'!//))   r;)rrrrry)r1r2r3r4rrirrr6r9r\r9rrr:rrs@r<rrs"=(/3)-,0#'&*C llT)C  t+C  $; C #Tk C D[ C  t#C  ) )C C r;rc^\rSrSrU4SjrSrSr\SS\RS-S\RS-S\RS-S \ S-S \ S-S \ S-S \ \ -4S jj5r SrU=r$)#Wav2Vec2ForAudioFrameClassificationi@c>[TU]U5 [US5(aUR(a [ S5e[ U5UlURS-nUR(a2[R"[R"U5U- 5Ul [R"URUR 5UlUR UlUR%5 g)NrzbAudio frame classification does not support the use of Wav2Vec2 adapters (config.add_adapter=True)r!)rrrrrRrYrrkrrrr6rarrrrrrdrs r<r,Wav2Vec2ForAudioFrameClassification.__init__Bs  6= ) )f.@.@t &f- --1  ( (!#ejj.Dz.Q!RD ))F$6$68I8IJ ++ r;cLURRR5 grgrrhs r<ri:Wav2Vec2ForAudioFrameClassification.freeze_feature_encoderRrr;cTURR5H nSUlM grrrs r<r5Wav2Vec2ForAudioFrameClassification.freeze_base_modelYrr;Nrr@rr(rprqrBc UbUOURRnURR(aSOUnURUUUUUS9nURR(aiU[n [ R "U SS9n [RRURSS9n XRSSS5-RSS9n OUSn URU 5n Sn UbZ[5n U "U RSUR5[ R "URSUR5SS95n U(dU 4U[S-nU$[#U U UR$UR&S 9$) rNTrur!rrNr)axisr)rrqrrrr6rrrrrr+rXrrrrrr,r-)rrr@rr(rprqrrZr,rrr(rrs r<r+Wav2Vec2ForAudioFrameClassification.forwardasf0&1%->r1a-HHMMRSMTM#AJM/  ')HFKKDOOtuDY)F)G!HHFM$!//))   r;)rrrrry)r1r2r3r4rrirrr6r9r\r9rrr:rrs@r<rr@s =(/3&*)-,0#': llT):  t+:  t# :  $; : #Tk : D[:  & &: : r;rc2^\rSrSrSU4SjjrSrSrU=r$) AMSoftmaxLossic>[TU]5 X0lX@lX l[ R "[R"X5SS9Ul [ R"5Ul g)NT)r) rrscalemarginrrrr6randnrrr()rrrrrrs r<rAMSoftmaxLoss.__init__sI   $ll5;;y#EUYZ '') r;cUR5n[RRURSS9n[RRUSS9n[ R "X5nX@R- n[RRX R5nUR[ R"UR5XT5-nURXr5nU$)Nrrr!)rrr normalizerr6mmrone_hotrrrr\r() rr,rr cos_thetapsionehotrr(s r<rAMSoftmaxLoss.forwards!((!(< // 1/E HH]3 ++%&&v?ekk&++-HHyy( r;)r(rrrr)g>@g?rrs@r<rrs*  r;rcf^\rSrSrSU4SjjrS\R S\R 4SjrSrU=r $) TDNNLayeric>[TU]5 US:aURUS- OURUUlURUUlUR UUlURUUl[R"URUR -UR5Ul [R"5Ul g)Nrr!)rrtdnn_dimrr tdnn_kernelr tdnn_dilationdilationrrkernelrrrs r<rTDNNLayer.__init__s [5(aSSKJn [5(a1[URW5(a[ R "S5 URSS5nURRRURURUR5RSS5n[RRXURR UR"S9nURSS5nUR%U5nU$)Nr) LoraLayerzDetected LoRA on TDNNLayer. LoRA weights won't be applied due to optimization. You should exclude TDNNLayer from LoRA's target modules.r!r#)r)rpeft.tuners.lorarrArwarningswarnrrr+rrrrrconv1drrr)rr,rrs r<rTDNNLayer.forwards    2   $++y11 O &//15 ##(():):D[TU]U5 [U5UlURS-nUR (a2[ R"[R"U5U- 5Ul [ R"URURS5Ul[[!UR55Vs/sHn[#X5PM nn[ R$"U5Ul[ R"URSS-UR(5Ul[ R"UR(UR(5Ul[/UR(UR05UlUR55 gs snf)Nr!rrNr#)rrrYrrkrrrr6rarrrrrrZr_rrtdnnxvector_output_dimr\rrr objectiverd)rrrr tdnn_layersrs r<rWav2Vec2ForXVector.__init__s  %f- --1  ( (!#ejj.Dz.Q!RD 6#5#5vq7IJ5:3v;O5PQ5Py+5P QMM+. !#6??2+>+BFD]D]!^))F$=$=v?X?XY&v'@'@&BSBST RsFcLURRR5 grgrrhs r<ri)Wav2Vec2ForXVector.freeze_feature_encoderrr;cTURR5H nSUlM grrrs r<r$Wav2Vec2ForXVector.freeze_base_modelrr;rjcXSnURRH nU"XS5nM U$)z/ Computes the output length of the TDNN layers cX- U-S-$)Nr!r0rs r<rEWav2Vec2ForXVector._get_tdnn_output_lengths.._conv_out_lengths!.69A= =r;r!)rr)rrjrrs r<_get_tdnn_output_lengths+Wav2Vec2ForXVector._get_tdnn_output_lengthss1  >  ;;22K,]KM3r;Nrr@r(rprqrrBc UbUOURRnURR(aSOUnURUUUUUS9nURR(aiU[n [ R "U SS9n [RRURSS9n XRSSS5-RSS9n OUSn URU 5n URH n U "U 5n M UcU RSS9n U R!SS9n OUR#URSS95nUR%U5n/n /n ['U5HPunnU R)U USU24RSS95 U R)U USU24R!SS95 MR [ R "U 5n [ R "U 5n [ R*"X/SS9nUR-U5nUR/U5nSnUbUR1UU5nU(dUU4U[S-nUbU4U-$U$[3UUUUR4UR6S9$) rNTrur!rrNr)r(r embeddingsr,r-)rrqrrrr6rrrrrr+rXrrrrr r enumeratercrr\rrrr,r-)rrr@r(rprqrrrZr,r tdnn_layer mean_features std_featuresfeat_extract_output_lengthstdnn_output_lengthsrlengthstatistic_poolingoutput_embeddingsrr(rs r<rWav2Vec2ForXVector.forward s0&1%->r1a-HHMMRSMTM#AJM}5 ))J&}5M$  !)..1.5M(,,,3L*.*O*OP^PbPbghPbPi*j '"&"?"?@["\ ML&':; 6$$]1gvg:%>%C%C%C%JK##M!WfW*$=$A$Aa$A$HI<"KK 6M ;;|4L!II}&CL 223DE!23  >>&&1D/07;X;Y3ZZF)-)9TGf$ Ev E(!//))   r;)rr\rrrrrry)r1r2r3r4rrirr6rWrFrrr9r\r9rrr:rrs@r<rrs &=( e6F6F6L /3)-,0#'&*P llT)P  t+P  $; P #Tk P D[ P  t#P   P P r;r)rrrrrrYrrrr[)br5rr collections.abcr dataclassesrnumpyrSr6safetensors.torchrr;rtorch.nnrr r activationsr integrations.deepspeedr integrations.fsdpr masking_utilsr modeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrrrmodeling_utilsrrrprocessing_utilsrrrrrrrrr configuration_wav2vec2r"r>r9 get_loggerr1r5rr&r9rFr8rWndarrayrqrrrrModulerrrrr9rrr;rLr]rgrrrrrarrYrrrrrrr__all__r0r;r<r>s $! 9%&!@76B9sr&3,5   H %!"  4;4  4B/3 t c?ttt$$t+ t  t ZZ tn$U$3$[][e[ehl[l$D#=*!;6!;0*bii*Z299%RYY%P 1 1,!% II% <<% % << % LL4' % T\ %%'(%8R/ R/j"))0!5!H+*D+\E biiE PI RYYI XI'BIII'Xbii>299$ryy2c'oc'c'L C +C C L N 4N  N b K ,K  K \ e (?e  e P[ *A[ [ |BII. @ C 0C  C L r;