Zj3SrSSKrSSKJr SSKrSSKJr SSKJrJrJ r SSK J r SSK Jr SS KJrJr SS KJr SS KJr SS KJr SS KJr SSKJrJrJr SSKJrJ r SSK!J"r" SSK#J$r$J%r%J&r&J'r' SSK(J)r) SSK*J+r+ SSK,J-r- SSK.J/r/ \'R`"\15r2"SS\Rf5r4S\RjS\6S\Rj4Sjr7S"S/S0\5r?\%"S1S2\ 55r@\%"S3S4\@55rA"S5S6\@\5rB\%"S7S89"S9S:\@55rC/S;QrDg)=zPyTorch Zamba model.N)Callable)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)lazy_load_kernel)create_causal_mask)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast SequenceClassifierOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)merge_with_config_defaults)resolve_internal_import)capture_outputs) ZambaConfigcx^\rSrSrS S\SS4U4SjjjrS\RS\R4SjrSr S r U=r $) ZambaRMSNorm1epsreturnNc>[TU]5 [R"[R "U55UlX lg)z+ ZambaRMSNorm is equivalent to T5LayerNorm N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizer# __class__s y/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/zamba/modeling_zamba.pyr'ZambaRMSNorm.__init__2s/ ll5::k#:; # hidden_statescURnUR[R5nUR S5R SSS9nU[R "X0R-5-nURURU5-$)NT)keepdim) dtypetor)float32powmeanrsqrtr,r+)r-r3 input_dtypevariances r0forwardZambaRMSNorm.forward:sw#)) %((7  $$Q',,R,>% H?T?T4T(UU {{]--k:::r2c^[URR5SUR3$)Nz, eps=)tupler+shaper,)r-s r0 extra_reprZambaRMSNorm.extra_reprAs* ))*+6$2G2G1HIIr2)r,r+)gư>) __name__ __module__ __qualname____firstlineno__floatr'r)Tensorr@rE__static_attributes__ __classcell__r/s@r0r!r!1sB$$$$$;U\\;ell;JJr2r!r3n_repr$cURup#pEUS:XaU$USS2SS2SSS2SS24RX#XU5nURX#U-XE5$)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rDexpandreshape)r3rPbatchnum_key_value_headsslenhead_dims r0 repeat_kvrXFs_ 2?1D1D.E z!!Qa"23::5W\dlmM  e(CT TTr2modulequerykeyvalueattention_maskscalingdropoutc [X R5n[X0R5n [R"XR SS55U-n UbX-n [ R RU S[RS9RUR5n [ R RXURS9n [R"X5n U R SS5R5n X4$)Nr5rr6)dimr8)ptrainingr)rXnum_key_value_groupsr)matmul transposer functionalsoftmaxr:r9r8r_rc contiguous) rYrZr[r\r]r^r_kwargs key_states value_states attn_weights attn_outputs r0eager_attention_forwardroRs3 ; ;[TU]5 XlX lURUlUR UlURUR-Ul URUl UR S- S-Ul SUl URUl [R"URURUR -SS9Ul[R"URURUR -SS9Ul[R"URURUR -SS9Ul[R"URUR -UR&SS9Ulg)Nr5TFbias)r&r'rsrtattention_hidden_sizeattention_head_dimrWnum_attention_headsrUrdmax_position_embeddingsr^ is_causalattention_dropoutrLinearq_projk_projv_projr.o_projr-rsrtr/s r0r'ZambaAttention.__init__ys5  "%+%A%A"11 $*$>$>&B\B\$\!'-'E'E$ )d2 !'!9!9ii < X>X[_[h[h>hotu ii < X>X[_[h[h>hotu ii < X>X[_[h[h>hotu ii : :T]] JFL^L^ejk r2Nr3r]past_key_valuesrjr$c URSSn/UQSPURP7nURU5RU5R SS5nUR U5RU5R SS5n UR U5RU5R SS5n UbURXU5up[R"URR[5n U "UUU U U4UR(dSO URURS.UD6upU R "/UQSP76R#5n UR%U 5n X4$)Nr6rr5)r_r^)rDrWrviewrfrrupdater get_interfacers_attn_implementationrorcr~r^rSrir)r-r3rtr]rrj input_shape hidden_shape query_statesrkrlattention_interfacernrms r0r@ZambaAttention.forwardsa$))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST  &'6'='=jXa'b $J(?(M(M KK , ,.E) %8      % $}}C$2H2HLL %  % ! "));;;;FFH kk+. ((r2) r~ryrsrWr}rrtr|rdrrr^rN)rGrHrIrJ__doc__rintr'r)rLr rrrCr@rMrNrOs@r0rqrqks l{lsl.)- #)||#)#) t+ #)  #) +, #) u||U\\D0% 2E2LL M#)#)r2rqc^\rSrSrSrS\4U4SjjrS S\RS\ S-4Sjjr S S\ S-4S jjr S S\ S-4S jjr S r U=r$) ZambaMambaMixeru! Compute ∆, A, B, C, and D the state space parameters and compute the `contextualized_states`. A, D are input independent (see Mamba paper [1] Section 3.5.2 "Interpretation of A" for why A isn't selective) ∆, B, C are input-dependent (this is a key difference between Mamba and the linear time invariant S4, and is why Mamba is called **selective** state spaces) This module differs from `transformers.models.mamba.modeling_mamba.MambaMixer` in two ways: - Added multi-head: the output of `self.in_proj` is split into `self.n_mamba_heads` heads, and each head undergoes an independent forward pass, identical to the original `MambaMixer`, up until the pre-activations of `self.out_proj`. The pre-activations, coming from different mamba heads, are then concatenated and fed into `self.out_proj`. rsc  >[TU]5 XlX lURUlUR UlURUlURUR-Ul URUl URUl URUR-UlURUlUR"Ul[&R("URURUR URURURS- S9UlUR,Ul[0UR,UlUR4Ul[&R8"URURS-UR$S9Ul[&R<"[>R@"URURUR S--UR55Ul![&R<"[>R@"URURUR5S- S-URS-- 5Ul"[&R<"[>R@"URUR55Ul#[>RH"SUR S-[>RJS9SSS24nURMURS5RO5n[&R<"[>RP"U5RSURURS55Ul*[&R<"[>RV"URUR55Ul,[&R8"URURUR$S9Ul-[]S5q/[a[^S S5q1[a[^S S5q2[]S 5q3[i[fS S 9q5[a[fSS5q6[a[fSS5q7[q[j[l[d[b[n45q9[r(d[tRwS5 gg)Nr) in_channels out_channelsrx kernel_sizegroupspaddingr5rwg?r8r6z causal-conv1dcausal_conv1d_updatecausal_conv1d_fnz mamba-ssmz8ops.triton.selective_state_update.selective_state_update) chained_pathselective_scan_fnmamba_inner_fnaqThe fast path is not available because one of `(selective_state_update, selective_scan_fn, causal_conv1d_fn, causal_conv1d_update, mamba_inner_fn)` is None. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d. If you want to use the naive implementation, set `use_mamba_kernels=False` in the model config)UR@UR@/SS9upnURBSS2S4U RSS5-n[(RD"URFRI55*nURJbURJRI5OSn[(RL"USU4URNURPS9nU(a[SUR5Hn[UURURRVSS2U4UUS S4UUS S4UUUUSS2S4UUSS2S4URXUU US S4UUS S 9 R'S5n[(RZ"UU4SS9nM GO0[(RL"USUR8UR@4URNURPS9n[SUR5Hn[]UUUUUUUURSS5UURSS5URXURI5U UUUS S S 9 unn[(RZ"UU4SS9R5n[(RZ"UUR'S54SS9nM UbUbUR_UUR5 URaURSS55nU$) Nrr5r6rar)rdevicer8.T) dt_softplus)delta_softplusreturn_last_state)1rDhas_previous_staterrfrchunksqueezerirSrrr+sizerlayersrt conv_statesrxr unsqueezer)rrrgpadrupdate_conv_staterrrsplitrrrexprrKremptyrr8rangerrecurrent_statesrcatrupdate_recurrent_stater)r-r3rr] batch_sizeseq_len_use_precomputed_statesprojected_statesgate conv_weightsrssm_parameters time_stepBCdiscrete_time_steprtime_proj_bias scan_outputsn scan_outputs_ ssm_state ssm_state_contextualized_statess r0cuda_kernels_forward$ZambaMambaMixer.cuda_kernels_forward sh"/!4!4 Q!-T!9!nl>]>]!nbimnbn << 6@@AF.33JAwOUUVW]^U_ %--a0;;= ||A||J(:(:BHRRSTVWX{{))..t{{/A/A/F/Fq/I4;;K]K]KbKbcdKef !0%%b)##DNN3??    M*33B7M)%))Na>7a<);;M>>Z % *{{7I7I!QPQ'7R*RXZ [ %%![[%5%55M $ 7 : :5 A K KB O !-$11}GZGZ[]G^F^F`C`4a4k4klm4n$nM]]..}?T?TWdWjWjkmWn?npq>rs );;JW $])CC'M)R S !-$11}GZGZ[]G^F^F`C`4a4k4klm4n$nM) -0H0H0K K  HHT[[%?XgX %NOM) -0H0H0K K &--b$2D2DdFYFY[bcmmnoqrs ,,Qa];mKVVWY[]^++ 00$2E2EtGZGZ[ac  a#11!T':Y=P=PQSUW=XX\`\m\m tQ ]   ]]334FGYYtzz'') * *YYqD!T1!458J1aQRTUW[K[8\\] '1aD(89AaD!Q>N ? E E GG wA"1aAq=1;;AqAIMPXYZ\]_`bcefYfPgPqPqrsuvPwwI,,y':':1a'@'C'CE'JAaQRTUWXjMLcLcdfLghK    Aq!QJ 7 8 kk,B7 !]VVAtQ> J!%  ! !!Q ' / / B H R RSTVW X! %$r2c [[[[[[ 45nUR (aFU(a$SURRR;a [S5eURXUS9$URXUS9$)NcudazFast Mamba kernels are not available. Make sure to they are installed and that the mamba module is on a CUDA device. lease run 'pip install causal-conv1d>=1.2.0' and 'pip install mamba-ssm', or set use_mamba_kernels=False in the model's config.)r]) rrrrrrrrrtype ValueErrorrr)r-r3rr]rjrs r0r@ZambaMambaMixer.forwards!$ #%68HJ^`n o"   )V4;M;M;T;T;Y;Y-Y i ,,]Yg,h h  ^ \\r2)rrrrrsrrrrr.rrrtrrrrrrrrr)NN)rGrHrIrJrrr'r)rLr rrr@rMrNrOs@r0rrsj M{M`^b_%"\\_%9>_%BP%ut|P%d ]54< ] ]r2rc.^\rSrSrU4SjrSrSrU=r$)ZambaMLPic>[TU]5 XlURUlURUl[ R "URURSS9Ul[ R "URURSS9Ul[ R "URURSS9Ul [URUl gNFrw) r&r'rsr.rrr gate_projup_proj down_projr hidden_actact_fnr-rsr/s r0r'ZambaMLP.__init__s  !--!'!9!94#3#3T5K5KRWXyy!1!143I3IPUV 4#9#94;K;KRWXV../ r2cURURURU55URU5-5nU$r)rrrr)r-xrs r0r@ZambaMLP.forwards6NN4;;t~~a/@#ADLLQRO#ST r2)rrsrrr.rr)rGrHrIrJr'r@rMrNrOs@r0rrs0r2rc^\rSrSrSS\S\S-4U4SjjjrSS\RS\RS\S\RS-S \ S-S \ S-S \ \ S \ \R\ \R\R4S-44S jjrSrU=r$)ZambaAttentionDecoderLayeriNrsrtc>[TU]5 [X5Ul[ U5Ul[ URURS9Ul [ URURS9Ul g)Nr#) r&r'rq self_attnr feed_forwardr!ry rms_norm_epsinput_layernormr.pre_ff_layernormrs r0r'#ZambaAttentionDecoderLayer.__init__s] ':$V,+F,H,HfNaNab ,V-?-?VEXEX Yr2r3original_hidden_statesr]r use_cacherjr$c [R"X/SS9nURU5nUR"SUUUUUS.UD6upUR U5nUR U5nU$)a Args: hidden_states (`torch.FloatTensor`): output of previous Mamba layer of shape `(batch, seq_len, embed_dim)` original_hidden_states (`torch.FloatTensor`): word embedding output of shape `(batch, seq_len, embed_dim)`. This is concatenated with `hidden_states` (which is the output of the previous (mamba) layer). The concatenated tensor is then used as input of the pre-attention RMSNorm (see fig. 2 in https://huggingface.co/papers/2405.16712). layer_idx (`int`): layer_idx in the forward pass. Used to distinguish Zamba's tied transformer layers. attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_values (`Cache`, *optional*): cached past key and value projection states use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). r6r)r3rtr]rr()r) concatenater$r!r%r") r-r3r'rtr]rr(rjrs r0r@"ZambaAttentionDecoderLayer.forwards2))=*QWYZ ,,]; >> ')+     --m< ))-8 r2)r"r$r%r!r)NNF)rGrHrIrJrrr'r)rLr boolrrrC FloatTensorr@rMrNrOs@r0rrsZ{ZsTzZZ/3(,!&'||'!& ' '  t+ '  '$;'+,' u  %(9(95;L;L(L"MPT"TT U''r2rc^\rSrSrS\S\4U4SjjrSS\RS\RS-S\S-S\RS-S \RS-S \ S-S \ S-S \RS-S \RS-S\ \ S\\R\\R\R4S-44SjjrSrU=r$)ZambaMambaDecoderLayerirsrtc>[TU]5 [XS9Ul[ UR UR S9UlX lg)N)rsrtr ) r&r'rmambar!r.r#r$rtrs r0r'ZambaMambaDecoderLayer.__init__s: $FH +F,>,>FDWDWX"r2Nr3r'r] causal_maskrr( position_idstransformer_hidden_statesrjr$c  rUn U bX-OUnURU5nUR"SUUUS.U D6nX-nU$)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_values (`Cache`, *optional*): cached past key and value projection states use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). )r3rr]r*)r$r2) r-r3r'rtr]r4rr(r5r6rjresiduals r0r@ZambaMambaDecoderLayer.forwardsg0! :S9^M 5dq ,,];   '()   !0 r2)r$rtr2)NNNNNFNN)rGrHrIrJrrr'r)rLr r- LongTensorrrrCr.r@rMrNrOs@r0r0r0s#{#s#7; $.2+/(,!&049=*||*!& t 3*: *  t+ * \\D( **$;*&&-*$)<<$#6*+,* u  %(9(95;L;L(L"MPT"TT U**r2r0cZ^\rSrSrS\S\R S\4U4SjjrSS\ RS\ RS-S \ S-S \ RS-S \ RS-S \ S-S \ S-S\\S\\ R"\\ R"\ R"4S-44SjjrSrU=r$)ZambaHybridLayeriH shared_transflinearr2cF>[TU]5 XlX lX0lgr)r&r'r=r> mamba_decoder)r-r=r>r2r/s r0r'ZambaHybridLayer.__init__Is * "r2Nr3r'rtr]r4rr(rjr$c UR"U4UUUUUS.UD6n URU 5n UR"U4U UUUS.UD6nU$)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` original_hidden_states (`torch.FloatTensor`): word embedding output that will be concatenated with hidden activations to form the input of the shared transformer layer. layer_idx (`int`): layer number. attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_values (`Cache`, *optional*): cached past key and value projection states use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). )r'rtr]rr()r6r]rr()r=r>r@) r-r3r'rtr]r4rr(rjr6s r0r@ZambaHybridLayer.forwardOs~2%)$6$6 % #9&+ % % !%)KK0I$J!**  &?)+    r2)r>r@r=)NNNNNF)rGrHrIrJrrrr0r'r)rLrr r-rrrCr.r@rMrNrOs@r0r<r<Hs#&@#"))#\r#7; $.2+/(,!&-||-!& t 3-: -  t+ - \\D( --$;-+,- u  %(9(95;L;L(L"MPT"TT U--r2r<c^\rSrSr%\\S'SrSrSS/rSr Sr Sr Sr \ \S .r\R""5U4S j5rS rU=r$) ZambaPreTrainedModelirsmodelTr<r0rF)r3 attentionsc>URRn[TU] U5 [ U[ 5(Ga[ R"URSUS9 URRS-n[ R"URU*U5 URRURR-URR-n[R "[R""URRU5[$R&"URR(5[$R&"URR*5- -[$R&"URR*5-5R-URR.S9nU[R&"[R0"U*5*5-n[ R2"UR4U5 [R6"SUR8S-[R:S9SSS24nUR=UR>S5RA5n[ R2"URB[R&"U5REURURFS55 [ RH"URJ5 gg)Nr)r<stdrv)minrrr6)&rsinitializer_ranger& _init_weights isinstancerinitnormal_rruniform_rrr.rr)rrandmathr time_step_max time_step_minclamptime_step_floorexpm1copy_rrrr:rRrrirrSrones_r) r-rYrI dt_init_stdrdtinv_dtrr/s r0rL"ZambaPreTrainedModel._init_weightsskk++ f% fo . . LL--CS A++33T9K MM&//+{ K![[55 8O8OOSWS^S^SlSllN 4;;44nE88DKK556$++B[B[9\\^((4;;4456e 33e4  %))U[["%5$566F JJv**F 3 Q 5 5 9OPTVWPWXA1126AACA JJv||UYYq\%9%9&:N:NPVPePegi%j k JJvxx % /r2r*)rGrHrIrJr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa _is_statefulr0rq_can_record_outputsr)no_gradrLrMrNrOs@r0rErEs_ &*#+-EF"3 NL/$  ]]_!!r2rEc ^\rSrSrSrS\4U4Sjjr\\\ SS\ RS-S\ RS-S\ RS-S \ S-S \ RS-S \S-S \\S \\-4Sjj555rSrU=r$) ZambaModeliz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`ZambaDecoderLayer`] Args: config: ZambaConfig rsc@>[TU]U5 URUlURUl[ R "URURUR5UlURUl /nSUl [UR5Hup4[XS9nUS:Xa[ R"URRURRSS9nUR[![#U5Xe55 URcSUS3SUS30Ul MMURU5 M [ R$"U5Ul[)URUR*S 9UlSUlUR15 g) N)rthybridFrwz layers.(?!z\.)\d+.shared_transfzlayers.z.shared_transfr )r&r' pad_token_id padding_idx vocab_sizer Embeddingr. embed_tokenslayers_block_type_tied_weights_keys enumerater0rrsrr<r ModuleListrr!r#final_layernormgradient_checkpointing post_init)r-rsrlayer_id layer_typer2r>r/s r0r'ZambaModel.__init__s]  !.. ++LL):):F,>FDWDWX&+# r2N input_idsr]r5r inputs_embedsr(rjr$c LUSLUSL- (a [S5eUcURU5nUn[R"U5n U(aUc[ UR S9nUcUUbUR 5OSn [R"URSURS9U -nURS5n[UR UUUUS9n [UR5HupU "UU U UU 4UUS.UD6nM URU5n[UU(aUS9$SS9$) NzaYou cannot specify both input_ids and inputs_embeds at the same time, and must specify either one)rsrrr)rsr|r]rr5)rr()last_hidden_stater)r rpr)rr rsget_seq_lengthrrDrrrrsrrur)r-r{r]r5rr|r(rjr3r'past_seen_tokensr4rtlayers r0r@ZambaModel.forwardsW  -t"; <s    --i8M% !&]!; 0*$++>O  CRC^==?de  << (;(;A(>}G[G[\_ooL'11!4L(;;')+%  !*$++ 6 I!&  !0#  M!7,,]; &+/8O  >B  r2)rrrprurvrrqrmrn)NNNNNN)rGrHrIrJrrr'rrrr)r:rLr r.r-rrrCrr@rMrNrOs@r0riris{8 .2.204(,26!%8 ##d*8  t+8 &&- 8  8 ((4/ 8 $;8 +,8  ( (8  8 r2ricb^\rSrSrSS0rS\4U4Sjjr\\SS\ RS-S\ RS-S \ RS-S \ S-S \ RS-S \ RS-S \S-S\\ R-S\\S\\-4Sjj55rSU4SjjrSrU=r$)ZambaForCausalLMi zlm_head.weightzmodel.embed_tokens.weightrsc>[TU]U5 [U5UlURUl[ R "URURSS9UlUR5 gr) r&r'rirFrnrrr.lm_headrwrs r0r'ZambaForCausalLM.__init__sU  '  ++yy!3!3V5F5FUS  r2Nr{r]r5rr|labelsr(logits_to_keeprjr$c hUR"SUUUUUUS.U D6n U Rn [U[5(a [ U*S5OUn UR U SS2U SS245n SnUbUR "U UUR40U D6n[UU U RU RU RS9$)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, ZambaForCausalLM >>> model = ZambaForCausalLM.from_pretrained("Zyphra/Zamba-7B-v1") >>> tokenizer = AutoTokenizer.from_pretrained("Zyphra/Zamba-7B-v1") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```)r{r]r5rr|r(Nlosslogitsrr3rGr*) rFrrMrslicer loss_functionrnrrr3rG)r-r{r]r5rr|rr(rrjoutputsr3 slice_indicesrrs r0r@ZambaForCausalLM.forwardsH,0::, )%+' , ,  11 8B>SV8W8W~ot4]k mA}a,?@A  %% D&#33!//))   r2c j>URRUS'[T U] "U4UUUUUUS.UD6n U $)Nr)rr]r|r5r(is_first_iteration)rsnum_logits_to_keepr&prepare_inputs_for_generation) r-r{rr]r|r5r(rrj model_inputsr/s r0r.ZambaForCausalLM.prepare_inputs_for_generation[sU$(;;#A#A w<   +)'%1    r2)rrFrn)NNNNNNNr)NNNNTF)rGrHrIrJrrrr'rrr)r:rLr r.r-rrrrCrr@rrMrNrOs@r0rr s0*,GH{.2.204(,26*.!%-.@ ##d*@  t+@ &&- @  @ ((4/ @   4'@ $;@ ell*@ +,@  ' '@ @ J r2ra The Zamba Model with a sequence classification head on top (linear layer). [`ZambaForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ) custom_introc^\rSrSrU4Sjr\\SS\RS-S\RS-S\RS-S\ S-S\RS-S \RS-S \ S-S \ \S \\-4S jj55rSrU=r$)ZambaForSequenceClassificationiuc>[TU]U5 URUl[U5Ul[ R "URURSS9UlUR5 gr) r&r' num_labelsrirFrrr.scorerwrs r0r''ZambaForSequenceClassification.__init__sS   ++' YYv114??O  r2Nr{r]r5rr|rr(rjr$c UR"U4UUUUUS.UD6n U Rn URU 5n UbURSn OURSn URR cU S:wa [ S5eURR cSn OUbXRR :gRU R[R5n[R"URSU R[RS9nX-RS5n O.Sn [RURR S35 U [R"XRS 9U 4nSnUGbURU R5nURR"cUR$S:XaS URlOoUR$S:aNUR&[R(:XdUR&[R*:XaS URlOS URlURR"S :XaJ[-5nUR$S:Xa&U"UR/5UR/55nOU"UU5nOURR"S :Xa=[15nU"UR3SUR$5UR3S55nO-URR"S :Xa[55nU"UU5n[7UUU R8U R:U R<S 9$)ae labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). )r]r5rr|r(Nrrz=Cannot handle batch sizes > 1 if no padding token is defined.r6rz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`r~ regressionsingle_label_classificationmulti_label_classificationr)rFrrrDrsrlr r9rr)int32rargmaxrrr/rG problem_typerr8longrrrrrrrrr3rG)r-r{r]r5rr|rr(rjtransformer_outputsr3rrlast_non_pad_token non_pad_mask token_indices pooled_logitsrloss_fcts r0r@&ZambaForSequenceClassification.forwards&8!F!Fr!J !#    >>**+,ZZ  u||J}}MOaab   YYv}}-F{{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#M$9$9$;V^^=MND#M6:D))-JJ+- 2 22t GUWY))-II,. v6/ /??-;;*55   r2)rFrr)NNNNNNN)rGrHrIrJr'rrr)r:rLr r.r-rrrCrr@rMrNrOs@r0rrus.2.204(,26*.!%R ##d*R  t+R &&- R  R ((4/ R   4'R $;R +,R  1 1R R r2r)rrrirE)r)ErrRcollections.abcrr)rtorch.nnrrrr rN activationsr cache_utilsr r generationr integrations.hub_kernelsr masking_utilsrmodeling_layersrmodeling_outputsrrrmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrutils.import_utilsrutils.output_capturingrconfiguration_zambar get_loggerrGrModuler!rLrrXrKrorqrrrr0r<rErirr__all__r*r2r0rs& $ AA&!.)8/9qqF&RR795,   H %J299J* UU\\ U# U%,, U&% II% <<% % << % LL4' %  %%2C)RYYC)L\]bii\]@ ryy  00f171h414n$!?$!$!N_ %_ _ Fg+_gT  ^ %9^  ^ B gr2