ZjM0SSKrSSKJr SSKrSSKrSSKJr SSKJr SSK J r SSK J r SSKJr SS KJr SS KJr SS KJr SS KJrJrJr SS KJrJrJr SSKJr SSK J!r!J"r"J#r# SSK$J%r% SSK&J'r' \#RP"\)5r*"SS\5r+"SS\5r,"SS\5r-"SS\R\5r/"SS\R\5r0"SS\R\5r1"SS \R\5r2SDS!\R\S"\RfS#\RfS$\RfS%\RfS-S&\4S-S'\4S(\\!4S)jjr5"S*S+\R\5r6"S,S-\R\5r7"S.S/\5r8"S0S1\R\5r9\""S2S3\55r:SES4\;\<\<4S5\4S6\S?\:55rB\""S@S=9"SASB\:55rC/SCQrDg)FN)Callable)nn)CrossEntropyLoss)initialization)ACT2FN)is_deepspeed_zero3_enabled)is_fsdp_managed_module)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputCausalLMOutputSequenceClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel*get_torch_context_manager_or_global_device)Unpack)TransformersKwargsauto_docstringlogging)is_flash_attention_requested) SEWConfigc2^\rSrSrSU4SjjrSrSrU=r$)SEWNoLayerNormConvLayer.cb>[TU]5 US:aURUS- OSUlURUUl[ R "URURURUURUURS9Ul [URUl g)Nrr kernel_sizestridebias)super__init__conv_dim in_conv_dim out_conv_dimrConv1d conv_kernel conv_stride conv_biasconvrfeat_extract_activation activationselfconfiglayer_id __class__s u/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/sew/modeling_sew.pyr# SEWNoLayerNormConvLayer.__init__/s [TU]5 US:aURUS- OSUlURUUl[ R "URURURUURUURS9Ul [ R"URSS9Ul [URUlg)NrrrT)elementwise_affine)r"r#r$r%r&rr'r(r)r*r+ LayerNorm layer_normrr,r-r.s r3r#SEWLayerNormConvLayer.__init__Ds [TU]5 US:aURUS- OSUlURUUl[ R "URURURUURUURS9Ul [URUl [ R"URURSS9Ulg)NrrrT) num_groups num_channelsaffine)r"r#r$r%r&rr'r(r)r*r+rr,r- GroupNormrKr.s r3r#SEWGroupNormConvLayer.__init___s [TU]5 [R"URURUR UR S-UR URS9Ul[RRn[[RRS5(a$[RRRn[5(Ga%SSKnURR!URR"SS9 U"URSSS9UlSSS5 [URS5(aUURRR"R$nURRR"R&nO,URR(nURR*nURR-X5 URR-X5 OU"URSSS9Ul[/UR 5Ul[2UR4Ulg!,(df  GN,=f) N)rpaddinggroupsr weight_normr modifier_rankweight)namedimparametrizations)r"r#rr' hidden_sizenum_conv_pos_embeddingsnum_conv_pos_embedding_groupssqueeze_factorr+utilsrehasattrrkr deepspeedzeroGatheredParametersrh original0 original1weight_gweight_vregister_external_parameterSEWSamePadLayerrcrr,r-)r/r0rerrrwrxr2s r3r##SEWPositionalConvEmbedding.__init__ws II      6622a777((   hh** 288,,m < <((33??K % ' ' 224993C3CST2U' aH Vtyy"4559955<<FF9955<<FF99--99-- NN 6 6t F NN 6 6t F#DIIH!DDI&v'E'EF  !?!?@VUs I I"clURU5nURU5nURU5nU$r7)r+rcr-r8s r3r:"SEWPositionalConvEmbedding.forwards2 -0  ]3  6 r5)r-r+rcr=rDs@r3r_r_vs ADr5r_c.^\rSrSrU4SjrSrSrU=r$)rzcR>[TU]5 US-S:XaSUlgSUlg)Nrbrr)r"r#num_pad_remove)r/rmr2s r3r#SEWSamePadLayer.__init__s) #:Q#>!#Car5cXURS:aUSS2SS2SUR*24nU$Nrrr8s r3r:SEWSamePadLayer.forwards6    ")!Q0F43F3F2F0F*FGMr5rr=rDs@r3rzrzsKr5rzc.^\rSrSrU4SjrSrSrU=r$) SEWUpsamplingc>[TU]5 [R"URURUR -5Ul[URUl UR Ulgr7) r"r#rLinearrlro projectionrr,r-r/r0r2s r3r#SEWUpsampling.__init__sW ))F$6$68J8JVMbMb8bc !?!?@$33r5c&URU5nURU5nURS:a^UR5up#nX0R-nX@R-nUR X#URU5nUR X%U5nU$)Nr)rr-rosizereshape)r/r9bszsrc_len src_embed_dimtgt_len tgt_embed_dims r3r:SEWUpsampling.forwards 6  6    "*7*<*<*> 'C- 3 33G)-@-@@M)11#@S@SUbcM)11# NMr5)r-rror=rDs@r3rrs4  r5rc8^\rSrSrSrU4SjrSrSrSrU=r $)SEWFeatureEncoderz.Construct the features from raw audio waveformc >[TU]5 URS:Xa@[USS9/[ UR S- 5Vs/sHn[ XS-S9PM sn-nOVURS:Xa-[ UR 5Vs/sH n[XS9PM nnO[SURS35e[R"U5Ul SUl S Ul gs snfs snf) Ngroupr)r1rlayerz`config.feat_extract_norm` is z), but has to be one of ['group', 'layer']FT)r"r#feat_extract_normrTrangenum_feat_extract_layersrrF ValueErrorr ModuleList conv_layersgradient_checkpointing_requires_grad)r/r0irr2s r3r#SEWFeatureEncoder.__init__s   # #w .0!DEINvOmOmpqOqIrIIrA'Q?IrIK % % 0NSTZTrTrNstNs0DNsKtK01I1I0JJst ==5&+#"Ius C C%cNUR5H nSUlM SUlgNF) parameters requires_gradrr/params r3_freeze_parameters$SEWFeatureEncoder._freeze_parameterss#__&E"'E '#r5cUSS2S4nUR(aUR(aSUlURH nU"U5nM U$)NT)rtrainingrr)r/ input_valuesr9 conv_layers r3r:SEWFeatureEncoder.forwardsK$QW-    4==*.M '**J&}5M+r5)rrr) r>r?r@rA__doc__r#rr:rBrCrDs@r3rrs8#"$  r5rmodulequerykeyvalueattention_maskscalingdropoutkwargsc UcURS5S-n[R"XRSS55U-nUbX-n[R R USS9n[R RXURS9n[R"X5n U RSS5R5n X4$)NrOrbrrj)prr) rtorchmatmulrPr functionalsoftmaxrr contiguous) rrrrrrrr attn_weights attn_outputs r3eager_attention_forwardrs**R.D(<<}}Q':;gEL!#4 ==((2(>L==((6??([L,,|3K''1-88:K  $$r5c:^\rSrSrSrSS\S\S\S\S\S \S \S-4U4S jjjr SS \ RS \ RS-S\ RS-S\S-S\ \ S\\ R\ RS-\\ RS-44 SjjrSrU=r$) SEWAttentioniz=Multi-headed attention from 'Attention Is All You Need' paperN embed_dim num_headsr is_decoderr! is_causalr0c>[TU]5 XlX lX0lX-UlXplUR U-UR:wa[SURSUS35eUR S-UlX@l X`l [R"XUS9Ul [R"XUS9Ul[R"XUS9Ul[R"XUS9Ulg)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r)r!)r"r#rrrhead_dimr0rrrrrrk_projv_projq_projout_proj) r/rrrrr!rr0r2s r3r#SEWAttention.__init__s "" !.  MMI %$.. 8MdnnM]$YKr3 }}d* $"ii 4@ ii 4@ ii 4@  )TB r5r9key_value_statesroutput_attentionsrreturnc USLnURSSn/UQSPURP7nURU5RU5R SS5n U(aUOUn /U RSSQSPURP7n UR U 5RU 5R SS5n UR U 5RU 5R SS5n [R"URR[5nU"UU U U U4UR(dSO URURUS.UD6unnUR"/UQSP76R!5nUR#U5nUUS4$)z#Input shape: Batch x Time x ChannelNrOrrb)rrr)shaperrviewrPrrr get_interfacer0_attn_implementationrrrrrrr)r/r9rrrris_cross_attention input_shape hidden_shape query_statescurrent_stateskv_shape key_states value_statesattention_interfacerrs r3r:SEWAttention.forward&s.T9$))#2. 88b8$--8 {{=166|DNNqRST -?)]B^))#2.BBDMMB[[055h?II!QO {{>277AKKAqQ (?(M(M KK , ,.E) %8      % $}}C$,,LL/ %  % ! \"));;;;FFH mmK0 L$..r5) r0rrrrrrrrrrr)rFTFN)NNF)r>r?r@rArintfloatboolrr#rTensorrr tupler:rBrCrDs@r3rrs G  #'CCC C  C  CCD CCD15.2). 0/||0/ ,,-0/ t+ 0/  $; 0/-.0/ u||U\\D0% 2E2LL M0/0/r5rc.^\rSrSrU4SjrSrSrU=r$)SEWFeedForwardiYc>[TU]5 [R"UR5Ul[R "URUR5Ul [UR[5(a[URUlOURUl[R "URUR5Ul[R"UR 5Ulgr7)r"r#rDropoutactivation_dropoutintermediate_dropoutrrlintermediate_sizeintermediate_dense isinstance hidden_actstrrintermediate_act_fn output_densehidden_dropoutoutput_dropoutrs r3r#SEWFeedForward.__init__Zs $&JJv/H/H$I!"$))F,>,>@X@X"Y f'' - -'-f.?.?'@D $'-'8'8D $IIf&>&>@R@RS jj)>)>?r5cURU5nURU5nURU5nURU5nUR U5nU$r7)rrrrrr8s r3r:SEWFeedForward.forwardgsX// > 00? 11-@ ))-8 ++M: r5)rrrrrr=rDs@r3rrYs @r5rc2^\rSrSrU4SjrSSjrSrU=r$)SEWEncoderLayeriqc>[TU]5 [URURUR SUS9Ul[R"UR5Ul [R"URURS9Ul [U5Ul[R"URURS9Ulg)NF)rrrrr0eps)r"r#rrlnum_attention_headsattention_dropout attentionrrrrrJlayer_norm_epsrKr feed_forwardfinal_layer_normrs r3r#SEWEncoderLayer.__init__rs %((00,,  zz&"7"78 ,,v'9'9v?T?TU*62 " V-?-?VEZEZ [r5cUnURXUS9upnURU5nXA-nURU5nXRU5-nUR U5nU4nU(aXu4- nU$)Nrr)rrrKr r )r/r9rr attn_residualr_outputss r3r:SEWEncoderLayer.forwards% )- L]*8* & Q ]3 %5  6 %(9(9-(HH --m<  "   &Gr5)rrr r rKrr=rDs@r3rrqs \r5rc:^\rSrSrU4SjrSSjrSrU=r$) SEWEncoderic>[TU]5 Xl[U5Ul[ R "URUR5Ul[ R"URURS9Ul [ R"UR5Ul[ R "[#UR$5Vs/sHn['U5PM sn5Ul[+U5UlSUlgs snf)NrF)r"r#r0r_pos_conv_embedr AvgPool1dropoolrJrlr rKrrrrrnum_hidden_layersrlayersrupsampler)r/r0rr2s r3r#SEWEncoder.__init__s  8@LL!6!68M8MN ,,v'9'9v?T?TUzz&"7"78 mmeFLdLdFe$fFe_V%RAUSSSXRS- 45nU(d[CS XU455$[EUUUS9$)NrOrrbrrdeviceg?dtype.rNNc3.# UH ocMUv M g7fr7r).0vs r3 %SEWEncoder.forward..sm$[q$[s last_hidden_stater9 attentions)# unsqueezerepeatrrr0longsumroraranger rexpandtor"finfominrPrrrrKrr r rrandr layerdroprrrpadrr )r/r9rroutput_hidden_states return_dictall_hidden_statesall_self_attentionsexpand_attention_mask input_lengthsoutput_lengthsmax_encoder_length attention_idsn_input_timestepsposition_embeddingspooled_hidden_states min_length synced_gpusrdropout_probabilityskip_the_layer layer_outputss r3r:SEWEncoder.forwards#7BD$5b4  %$2$<$Q>Q2R2V2V!V!/!6!6"((+Q0D0DR0H.J^J^_aJb"*//2%//15 "11-@#yy7,11"57K7P7PQS7TU ,S+:+-=>ATUXZe[eZeUeAff %//15  6  ]3 02R6LT6R [[E#$58H$H!#(**R. !]]Z/BT[[EZEZ/ZN![ %!Te! !.a 0  ,   &9]1=M'>q'AFKKD[D['["\]]  NN6;;++Q /  * * LLSdkk6S6S T r|| < = = KK $ JJv}} %  * *)++ 6:..76:3N3N"::FOOV__;]mn:o,,V]];po#::6==XY:Z,,V]];[Z$$V]]3 fryy"))4 5 5&++:Q KK $;R 5po[Zs?!K6 !L6 L Lr=cSn[URRURR5Hup4U"XU5nM U$)z8 Computes the output length of the convolutional layers c8[R"X- USS9S-$)Nfloor) rounding_moder)rdiv) input_lengthrr s r3_conv_out_lengthMSEWPreTrainedModel._get_feat_extract_output_lengths.._conv_out_length*s 99\7wWZ[[ [r5)zipr0r(r))r/r=rcrr s r3 _get_feat_extract_output_lengths3SEWPreTrainedModel._get_feat_extract_output_lengths%sG  \ $'t{{'>'> @W@W#X K,]PM$Yr5feature_vector_lengthrcURURS55R[R5nUR Sn[R "XA4URURS9nSU[R"UR SURS9US- 4'URS/5RS5RS/5R5nU$)NrOr)r"r rr) rfr/r2rr.rzerosr"r r0flipcumsumr)r/rhrr> batch_sizes r3"_get_feature_vector_attention_mask5SEWPreTrainedModel._get_feature_vector_attention_mask4s>>~?Q?QRT?UVYYZ_ZdZde#))!,   /~7K7KTbTiTi uv ^%9%9!%<^EZEZ[]kno]opq',,bT299"=BBB4HMMOr5rN)r>r?r@rAr__annotations__base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attnrno_gradr[ LongTensorrrfrnrBrr5r3rKrKsz $O&*#N ]]_%%< e>N>NQT>T   ]b]m]m r5rKr mask_prob mask_length min_masksrc ^^^^^UunmTS:a [S5eTT:a[STSTS35e[RRS5R 5mUUUUU4SjnUb-UR 5R S5R5O[U5Vs/sHnTPM snn[R"UT4[S 9n /n U"T5n U S :XaU $UHn U"U 5n [RR[R"U TS- - 5U S S 9n[U5S :XaTS- nOUS n[R"U[R"X- [R S 9U-/5nU R#U5 M [R$"U 5n [R&"U SS2SS2S4X[T45n U R)X[T-5n [R"T5SSSS24n[R&"UX[T45R)X[T-5nU U-n U R+5TS- :a TS- XTS- :'[R,"XSS5 U $s snf) a2 Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://huggingface.co/papers/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. rz&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `c>[TU-T- T-5n[UT5nUT-T:aTT-nUTS- - U:a[UTS- - S5nU$)z;Given input length, compute how many spans should be maskedrr)rmax)rbnum_masked_spanepsilonr{rzr|sequence_lengths r3compute_num_masked_span6_compute_mask_indices..compute_num_masked_spangsoi,6DwNOoy9 [ (? :-[TU]U5 Xl[U5Ul[ R "URSURS9Ul URSUR:gUl UR(a3[ R"URSUR5Ul [ R"UR5UlUR"S:dUR$S:aG[ R&"[(R*"UR5R-55Ul[1U5UlUR55 g)NrOrr)r"r#r0rfeature_extractorrrJr$r rKrlproject_featuresrfeature_projectionrfeat_proj_dropoutfeature_dropoutmask_time_probmask_feature_prob Parameterrruniform_masked_spec_embedrencoder post_initrs r3r#SEWModel.__init__s   !26!:,,vr':@U@UV & 3v7I7I I  &(ii0CVEWEW&XD #!zz&*B*BC  3 &&*B*BS*H%'\\%,,v?Q?Q2R2[2[2]%^D "!&)  r5Nr9mask_time_indicesrc[URSS5(dU$UR5upEnUb(URR UR 5X'OURR S:aUR(a[XE4URR URRUURRS9n[R"X!R[RS9nURR UR 5X'URRS:aUR(a[XF4URRURR URR"S9n[R"XqR[RS9nUSS2S4R%SUS5nSX'U$) z Masks extracted features along time axis and/or along feature axis according to [SpecAugment](https://huggingface.co/papers/1904.08779). apply_spec_augmentTNr)rzr{rr|)r r")rzr{r|rO)getattrr0rrr2r"rrrmask_time_lengthmask_time_min_masksrtensorr rrmask_feature_lengthmask_feature_min_masksr1)r/r9rrrmrrlmask_feature_indicess r3_mask_hidden_statesSEWModel._mask_hidden_statesst{{$8$?? 4A3E3E3G0 [  (/3/E/E/H/HI\I\/]M , [[ ' '! + 5-++44 KK88-++99 ! !& ->G[G[chcmcm n /3/E/E/H/HI\I\/]M , ;; ( (1 ,#8)++77 KK;;++<< $ $)<<0DMaMainisis#t #74#@#G#GO]_#` 23M /r5rrr8r9rc bUbUOURRnUbUOURRnUbUOURRnUR U5nUR SS5nUR U5nUR(aURU5nURU5n UbURU RSU5nURXS9n URU UUUUS9n U Sn U(d U 4U SS-$[U U RU R S9$)a mask_time_indices (`torch.BoolTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices to mask extracted features for contrastive loss. When in training mode, model learns to predict masked extracted features in *config.proj_codevector_dim* space. Nrrb)rrrr8r9rr))r0rr8r9rrPrKrrrrnrrrr r9r+) r/rrrrr8r9rextract_featuresr9encoder_outputss r3r:SEWModel.forwardsR 2C1N-TXT_T_TqTq$8$D $++JjJj &1%  %!DD]EXEXYZE[]klN000d ,, )/!5# ' (* !#oab&99 9+)77&11  r5)r0rrrrrKrrr#NNNNN)r>r?r@rArr#r FloatTensorryrrrrrr r:rBrCrDs@r3rrsy.7;26 ,((,!,,t3,((4/ ,\/36:)-,0#'4 llT)4  t+4 !,,t3 4  $; 4 #Tk 4 D[4   4 4 r5rzk SEW Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC). ) custom_introc^\rSrSrSS\S-4U4SjjjrSrSrSr\ SS\ RS-S \ RS-S \ S-S \ S-S \ S-S \ RS-S\ \-4Sjj5rSrU=r$) SEWForCTCi7N target_langc>[TU]U5 [U5Ul[R "UR 5UlX lURc[SURS35e[US5(aUR(a URO URn[R "X1R5UlUR%5 g)a target_lang (`str`, *optional*): Language id of adapter weights. Adapter weights are stored in the format adapter..safetensors or adapter..bin. Only relevant when using an instance of [`SEWForCTC`] with adapters. Uses 'eng' by default. NzYou are trying to instantiate z with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `SEWForCTC.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration. add_adapter)r"r#rrLrr final_dropoutrr vocab_sizerr2rqroutput_hidden_sizerlrlm_headr)r/r0rrr2s r3r#SEWForCTC.__init__=s  F#zz&"6"67 &    $00@AHH *1)G)GFL^L^F % %djdvdv yy!35F5FG  r5c @[5[R"S5:XagURnUb'[ UR SS5c[ SUS35eUc.[ UR SS5b[RS5 gUbURUSS9 gg) a This method overwrites [`~PreTrainedModel.tie_weights`] so that adapter weights can be correctly loaded when passing `target_lang=...` to `from_pretrained(...)`. This method is **not** supposed to be called by the user and is prone to be changed in the future. metaNadapter_attn_dimzCannot pass `target_lang`: z- if `config.adapter_attn_dim` is not defined.z)By default `target_lang` is set to 'eng'.T) force_load) rrr rrr0rloggerinfo load_adapter)r/rrs r3 tie_weightsSEWForCTC.tie_weightsZs 6 75<<;O O  &&  "wt{{r?r@rArr#rrrrrrrrrr:rBrCrDs@r3rr7s C$J:<08(/3)-,0#'&*E llT)E  t+E  $; E #Tk E D[ E  t#E   E E r5rz SEW Model with a sequence classification head on top (a linear layer over the pooled output) for tasks like SUPERB Keyword Spotting. c^\rSrSrU4SjrSrSr\SS\RS-S\RS-S\ S-S \ S-S \ S-S \RS-S \ \ -4S jj5r SrU=r$)SEWForSequenceClassificationic">[TU]U5 [US5(aUR(a [ S5e[ U5UlURS-nUR(a2[R"[R"U5U- 5Ul [R"URUR 5Ul[R"UR UR$5UlUR)5 g)NrzZSequence classification does not support the use of SEW adapters (config.add_adapter=True)r)r"r#rqrrrrLruse_weighted_layer_sumrrrr layer_weightsrrlclassifier_proj_size projector num_labels classifierr)r/r0 num_layersr2s r3r#%SEWForSequenceClassification.__init__s  6= ) )f.@.@l F#--1  ( (!#ejj.Dz.Q!RD 6#5#5v7R7RS))F$?$?ARARS r5cLURRR5 grrrs r3r3SEWForSequenceClassification.freeze_feature_encoderrr5cTURR5H nSUlM grrrs r3r.SEWForSequenceClassification.freeze_base_modelrr5Nrrrr8r9rrc 0UbUOURRnURR(aSOUnURUUUUUS9nURR(aiU[n [ R "U SS9n [RRURSS9n XRSSS5-RSS9n OUSn URU 5n UcU RSS9n OURU R SU5n U R#S5R%SSU R S5n S X)'U RSS9U RSS9RSS5- n UR'U 5nSnUbF[)5nU"URSURR*5URS55nU(dU4U[S-nUbU4U-$U$[-UUUR.UR0S 9$) a input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoProcessor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`SEWProcessor.__call__`] for details. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NTrrrrOrrbrr)r0r9rrLrrstackrrrrrr/rrOrnrr,r-rrrrr9r+)r/rrrr8r9rrrr9 norm_weights pooled_output padding_maskexpand_padding_maskrrloss_fctrs r3r:$SEWForSequenceClassification.forwards0&1%->r1a-HHMMRSMTM#AJM}5  !)..1.5MBB=CVCVWXCY[ijL"."8"8"<"C"CAq-J]J]^_J`"a 25M. /)--!-4|7G7GA7G7N7S7STVXY7ZZM/  ')HFKKDKK,B,BCV[[QS_UDY)F)G!HHF)-)9TGf$ Ev E'!//))   r5)rrrrLr)r>r?r@rAr#rrrrrrrrr:rBrCrDs@r3rrs"8(/3)-,0#'&*C llT)C  t+C  $; C #Tk C D[ C  t#C  ) )C C r5r)rrrrK)Nrr)ErScollections.abcrnumpyrrrtorch.nnrrrQ activationsrintegrations.deepspeedr integrations.fsdpr modeling_flash_attention_utilsr modeling_layersr modeling_outputsr rrmodeling_utilsrrrprocessing_utilsrrprrr utils.genericrconfiguration_sewr get_loggerr>rrrFrTModuler_rzrrrrrrrrrrKrrryndarrayrrrrr__all__rr5r3r$s* $ %&!@7B9YYrr&@@9(   H %8*6660((VbiiBII,# #X!% II% <<% % << % LL4' % T\ %%'(%8R/299R/jRYY0!0!Hc c LBBBR/3 t c?ttt$$t+ t  t ZZ tnx !x x v!" K "K  K \ e #5e  e P Zr5