ZjsSSKJr SSKJr SSKrSSKJr SSKJr SSK J r SSK J r J r SS KJr SS KJrJrJrJr SS KJrJr SS KJr SS KJrJr SSKJrJr SSK J!r!J"r" SSK#J$r$ SSK%J&r&J'r'J(r(J)r) SSK*J+r+J,r, SSK-J.r.J/r/ SSK0J1r1 "SS\Rd5r3\"S5"SS\Rd55r4"SS\Rd5r5"SS\Rd5r6\"SS \Rd55r7"S!S"\Rd5r8S#r9\"S$5S>S%j5r:S&\RvS'\S0\>S1\$\&4S2jjr?\"\:5"S3S4\Rd55r@"S5S6\5rA"S7S8\"5rB\'"S9S:\B55rC\'"S;S<\B\55rD/S=QrEg)@)Callable)OptionalN)nn)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)use_experts_implementationuse_kernel_forward_from_hubuse_kernel_func_from_hubuse_kernelized_func)create_causal_mask!create_sliding_window_causal_mask)GradientCheckpointingLayer)MoeCausalLMOutputWithPastMoeModelOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleis_grouped_mm_available)maybe_autocastmerge_with_config_defaults)OutputRecordercapture_outputs) AfmoeConfigc ^\rSrSr%\R \S'SS\4U4Sjjjr\ SS\S-S\ SS\ S-S \ S \ 44S jj5r\R"5\S 55rS rU=r$)AfmoeRotaryEmbedding1inv_freqNconfigc>[TU]5 URUlURUlXlUR R SUlURnURS:wa[URnU"UR U5uo@l URSUSS9 URSUR5SS9 g)N rope_typedefaultr'F) persistentoriginal_inv_freq) super__init__max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr(rope_parametersr*compute_default_rope_parametersrattention_scalingregister_bufferclone)selfr(device rope_init_fnr' __class__s y/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/afmoe/modeling_afmoe.pyr/AfmoeRotaryEmbedding.__init__4s "("@"@$*$B$B! 44[A!%!E!E >>Y &.t~~>L+7 V+L(( ZeD 0(..2BuUr9z torch.deviceseq_lenreturnz torch.Tensorc URSn[USS5=(d URUR-nSnSU[R "SUS[R S9RU[RS9U- -- nXe4$) aH Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetahead_dimNg?rdtype)r9rF) r3getattr hidden_sizenum_attention_headstorcharangeint64tofloat)r(r9r?basedimattention_factorr's r<r44AfmoeRotaryEmbedding.compute_default_rope_parametersDs&%%l3fj$/c63E3EIcIc3c U\\!S!5;;?BB&X]XcXcBdgjj k ))r>cLURSSS2S4R5RURSSS5R UR 5nUSS2SSS24R5n[ UR R[5(a0UR RS:waUR ROSn[USS9 UR5UR5-RSS5n[R"Xf4SS 9nUR5UR-nUR5UR-n SSS5 WR UR S 9W R UR S 94$!,(df  N@=f) Nrr"mpscpuF) device_typeenabledrDrPrE)r'rNexpandshaperMr9 isinstancetypestrr transposerJcatcosr5sinrF) r8x position_idsinv_freq_expandedposition_ids_expandedrWfreqsembrarbs r<forwardAfmoeRotaryEmbedding.forwardbsN!MM$4-8>>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E'E!((--[`J`ahhmmfk  U C&,,.1F1L1L1NNYYZ[]^_E))UN3C'')d444C'')d444C D vvAGGv$cff177f&;;; D Cs BF F#)r5r(r1r2r*N)NNN)__name__ __module__ __qualname____firstlineno__rJTensor__annotations__r#r/ staticmethodrinttuplerNr4no_gradrri__static_attributes__ __classcell__r;s@r<r%r%1sllV{VV %)+/"*d"*(*t* ~u$ % **: ]]_ < r%RMSNormc`^\rSrSrS S\SS4U4SjjjrS\R4SjrSr Sr U=r $) AfmoeRMSNormrepsr@Nc>[TU]5 [R"[R "U55UlX lg)z+ AfmoeRMSNorm is equivalent to T5LayerNorm N)r.r/r ParameterrJonesweightvariance_epsilon)r8rHr}r;s r<r/AfmoeRMSNorm.__init__ts/ ll5::k#:; #r>cURnUR[R5nUR S5R SSS9nU[R "X0R-5-nURU-RU5$)NrDrTT)keepdim) rFrMrJfloat32powmeanrsqrtrr)r8 hidden_states input_dtypevariances r<riAfmoeRMSNorm.forward|sw#)) %((7  $$Q',,R,>% H?T?T4T(UU  m+// <c^[URR5SUR3$)Nz, eps=)rtrr[r)r8s r< extra_reprAfmoeRMSNorm.extra_reprs* ))*+6$2G2G1HIIr>)rr)gư>) rlrmrnrorNr/rJrprirrvrwrxs@r<r{r{rs7$$$$$= =JJr>r{c2^\rSrSrSU4SjjrSrSrU=r$)AfmoeMLPc>[TU]5 XlURUlUc UROUUl[ R "URURSS9Ul[ R "URURSS9Ul[ R "URURSS9Ul [URUl gNFbias) r.r/r(rHintermediate_sizerLinear gate_projup_proj down_projr hidden_actact_fn)r8r(rr;s r<r/AfmoeMLP.__init__s  !--=N=V!9!9\m4#3#3T5K5KRWXyy!1!143I3IPUV 4#9#94;K;KRWXV../ r>cURURURU55URU5-5nU$rk)rrrr)r8rcrs r<riAfmoeMLP.forwards6NN4;;t~~a/@#ADLLQRO#ST r>)rr(rrrHrrrk)rlrmrnror/rirvrwrxs@r<rrs0r>rcf^\rSrSrSrU4SjrS\RS\R4SjrSr U=r $)AfmoeTokenChoiceRouterz Token-choice top-K router for MoE routing. This router assigns each token to the top-K experts based on sigmoid scores, matching the released checkpoints. c>[TU]5 XlURUlUR UlUR Ul[R"URUR SS9Ul gr) r.r/r(num_experts_per_toktop_k num_experts route_scalerrrHgater8r(r;s r<r/AfmoeTokenChoiceRouter.__init__s\  // !--!--IIf00&2D2D5Q r>r expert_biascURu p4URSU5nURU5R[R 5n[R "U5n[R"Xb-URSS9up7URSUS9nURSSS9S-n X- nXR-nXXU4$)NrTr")krP)rPindexT)rPrg#B ;) r[viewrrMrJrsigmoidtopkrgathersumr) r8rr_ hidden_dim router_logitsscoresselected_experts top_scores denominators r<riAfmoeTokenChoiceRouter.forwards(..1%**2z:  -033EMMB }-#jj)=QRS]]q0@]A  nnTn:UB - "2"22 *:::r>)r(rrrr rlrmrnro__doc__r/rJrprirvrwrxs@r<rrs. R ;U\\ ; ; ;r>rc^\rSrSrSrU4SjrS\RS\RS\RS\R4SjrS r U=r $) AfmoeExpertsz2Collection of expert weights stored as 3D tensors.c>[TU]5 URUlURUlUR Ul[R"[R"URSUR -UR55Ul [R"[R"URURUR 55Ul [URUlg)NrD)r.r/rrHrmoe_intermediate_sizeintermediate_dimrrrJempty gate_up_projrrrrrs r<r/AfmoeExperts.__init__s !-- ,, & < <LLT5E5Eq4K`K`G`bfbqbq)rsekk$2B2BDOOUYUjUj&klV../ r>r top_k_index top_k_weightsr@cX[R"U5n[R"5 [RRR X R S9nURSSS5n[R"URSS9S5R5nSSS5 WHnUSnXpR :XaM[R"WU5upXn [RRXRU5RSSS9upURU 5U -n [RRXR U5n XXS4-n UR#SXR%UR&55 M U$!,(df  N=f)N) num_classesrDr"r)rTrYrT)rJ zeros_likerur functionalone_hotrpermutegreaterrnonzerowherelinearrchunkrr index_add_rMrF)r8rrrfinal_hidden_states expert_mask expert_hit expert_idx top_k_pos token_idx current_staterupcurrent_hidden_statess r<riAfmoeExperts.forwardso $..}= ]]_((--55kO_O_5`K%--aA6K{8'DaHPPRJ %J#AJ---#(;;{:/F#G I)4M}}++M;L;LZ;XY__`agi_jHD$(KK$5$: !$&MM$8$89NP^P^_iPj$k !$9)`dJdrc2^\rSrSrSrU4SjrSrSrU=r$)AfmoeSparseMoeBlockz Mixture of Experts (MoE) module for AFMoE. This module implements a sparse MoE layer with both shared experts (always active) and routed experts (activated based on token-choice routing). c,>[TU]5 Xl[U5Ul[ XR UR-5Ul[U5Ul [R"[R"UR5SS9Ulg)NF) requires_grad)r.r/r(rrouterrrnum_shared_expertsshared_expertsrexpertsrrrJzerosrrrs r<r/AfmoeSparseMoeBlock.__init__sl  ,V4 &v/K/KfNgNg/gh#F+ << F4F4F(GW\]r>cURup#nURSU5nURXR5upgnUR U5RX#U5n UR XXU5RX#U5n X-$)NrT)r[rrrrr) r8r batch_sizer?rhidden_states_flatrrr shared_output routed_outputs r<riAfmoeSparseMoeBlock.forwards*7*=*=' Z*//J?7;kk-QaQa6b3 #3++,>?DDZZde  %7:V[[  ,,r>)r(rrrr) rlrmrnrorr/rirvrwrxs@r<rrs^ - -r>rcUSSURSS-24nUSURSS-S24n[R"U*U4SS9$)z*Rotates half the hidden dims of the input..NrTrDrY)r[rJr`)rcx1x2s r< rotate_halfrsZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''r>rotary_pos_embcURU5nURU5nX-[U5U--nX-[U5U--nXV4$)aIApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer)qrrarb unsqueeze_dimq_embedk_embeds r<apply_rotary_pos_embrsS& -- &C -- &Cw;q>C/0Gw;q>C/0G  r>rn_repr@cURup#pEUS:XaU$USS2SS2SSS2SS24RX#XU5nURX#U-XE5$)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r"N)r[rZreshape)rrbatchnum_key_value_headsslenrCs r< repeat_kvrs_ 2?1D1D.E z!!Qa"23::5W\dlmM  e(CT TTr>modulequerykeyvalueattention_maskscalingdropoutkwargsc [X R5n[X0R5n [R"XR SS55U-n UbX-n [ R RU S[RS9RUR5n [ R RXURS9n [R"X5n U R SS5R5n X4$)NrDrrT)rPrF)ptrainingr")rnum_key_value_groupsrJmatmulr_rrsoftmaxrrMrFr r contiguous) rrrr r r r r  key_states value_states attn_weights attn_outputs r<eager_attention_forwardr)s3 ; ;c^\rSrSrSrS\S\4U4SjjrSS\RS\ \R\R4S \RS-S \ S-S \ \ S \ \R\R44 S jjrSrU=r$)AfmoeAttentioniBa6 Multi-headed attention module with optional sliding window and gating. This attention mechanism supports both full attention and sliding window attention, and includes Q/K normalization and gating of the output. It inherits from [`LlamaAttention`] to minimize the amount of custom logic we need to maintain. r( layer_idxc>[TU]5 XlX l[ USUR UR -5UlUR UR-Ul URS-Ul URUl SUl [R"UR UR UR-URS9Ul[R"UR URUR-URS9Ul[R"UR URUR-URS9Ul[R"UR UR-UR URS9UlUR(US:HUlUR*(a UR,OSUl[/URUR0S9Ul[/URUR0S9Ul[R"UR UR UR-SS9Ulg)NrCgTrsliding_attentionr}F)r.r/r(rrGrHrIrCrrr attention_dropout is_causalrrattention_biasq_projk_projv_projo_proj layer_typesis_local_attentionsliding_windowr{ rms_norm_epsq_normk_normrr8r(rr;s r<r/AfmoeAttention.__init__Ls  " F4F4F&JdJd4de $*$>$>&B\B\$\!}}d* !'!9!9ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii  & & 68J8JQWQfQf  #)"4"4Y"?CV"V7;7N7Nf33TX"4==f6I6IJ "4==f6I6IJ 6#5#5v7Q7QTXTaTa7ahmnr>Nrposition_embeddingsr past_key_valuer r@c URSSn/UQSPURP7nURU5RU5nUR U5RU5n UR U5RU5n UR U5n URU5RSS5nURU 5RSS5n U RSS5n UR(aUup[XX5upUbURXUR5up[R"UR R"[$5nU"UUU U 4UUR&(dSO UR(UR*UR,S.UD6unnUR"/UQSP76R/5nU[0R2"U 5-nUR5U5nUU4$)NrTr"rD)r r r r))r[rCr#rr$r%rr+r_r,r(rupdaterr get_interfacer(_attn_implementationrrr r r)rrJrr&)r8rr/r r0r  input_shape hidden_shape query_statesrr gate_statesrarbattention_interfaceoutputrrs r<riAfmoeAttention.forwardks$))#2. 88b8$--8 {{=166|D [[/44\B {{=166|D nn]3 {{<0::1a@ [[,66q!< #--a3  " "*HC';LVY'_ $L  %'5'<')r r(rrCr!r(r,r$rrr&r+r#r r)r%rk)rlrmrnrorr#rsr/rJrprtr rrrirvrwrxs@r<rrBso{osoH(, .)||.)#5<<#=>.) t+ .)  .) +, .) u||U\\) *.).)r>rc^\rSrSrSrS\S\4U4SjjrSS\RS\RS-S \RS-S \ S-S \ S-S \ \R\R4S-S \\S\R 4SjjrSrU=r$)AfmoeDecoderLayeriz AFMoE decoder layer with dual normalization. This layer applies self-attention followed by either a dense MLP or MoE block, with dual normalization (pre and post) around each component. r(rc>[TU]5 URUlX l[ XS9Ul[ URURS9Ul[ URURS9Ul [ URURS9Ul [ URURS9Ul X!R:Ul UR(a[U5Ulg[!U5Ulg)N)r(rr)r.r/rHrr self_attnr{r*input_layernormpost_attention_layernormpre_mlp_layernormpost_mlp_layernormnum_dense_layers moe_enabledrmlprr-s r<r/AfmoeDecoderLayer.__init__s !--"'vK ,F,>,>FDWDWX(4V5G5GVM`M`(a%".f.@.@fFYFY!Z".v/A/AvGZGZ"[%(?(??   *62DH'DHr>Nrr rdr0 use_cacher/r r@c UnURU5nUR"SUUUUUUS.UD6upURU5nX-nUnURU5nUR U5nUR U5nX-nU$)N)rr rdr0rIr/)rAr@rBrCrGrD) r8rr rdr0rIr/r residualrs r<riAfmoeDecoderLayer.forwards!,,]; >> ')%) 3    55mD  0 !..}= / // >  0 r>) rHrArrGrFrBrDrCr@)NNNNN)rlrmrnrorr#rsr/rJrp LongTensorr boolrtrr FloatTensorrirvrwrxs@r<r>r>s({(s(2/304'+!%HL!||! t+!&&- !  ! $; !#5<<#=>E!+,!   !!r>r>c^\rSrSr%Sr\\S'SrS/rS/r \ "\ SS9\ \ S .r/S QrS rS rS r\"5rS rS rU4S jrS rU=r$)AfmoePreTrainedModelizz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. r(modelr>past_key_valuesr)r)rr attentions)rArBrCrDr+r,normrTc>[TU]U5 URRn[ U[ 5(aA[ R"URSUS9 [ R"URSUS9 g[ U[5(a+[ R"URR5 g[ U[5(a![ R"UR5 gg)zInitialize the weightsr2)rstdN)r. _init_weightsr(initializer_ranger\rinitnormal_rrrzeros_rrrr)r8rrXr;s r<rY"AfmoePreTrainedModel._init_weightss f%kk++ fl + + LL,,3C @ LL)) =  6 7 7 KK ** +  3 4 4 KK** +5r>rK)rlrmrnrorr#rqbase_model_prefix_no_split_modules_skip_keys_device_placementr rr>r_can_record_outputs_keep_in_fp32_modules_supports_sdpa_supports_flash_attn_supports_flex_attnr_can_compile_fullgraph_supports_attention_backendsupports_gradient_checkpointingrYrvrwrxs@r<rRrRs ,-#4"5'(>aH*$ N!#'&*# , ,r>rRc ^\rSrSrSrS\4U4Sjjr\\\ SS\ RS-S\ RS-S\ RS-S \ RS-S \S-S \S-S \\S \\-4Sjj555rSrU=r$) AfmoeModeli z Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`AfmoeDecoderLayer`] Args: config: AfmoeConfig r(c >[TU]U5 URUlURUl[ R "URURUR5Ul[ R"[UR5Vs/sHn[X5PM sn5Ul [URURS9Ul[#US9UlSUlUR)5 gs snf)Nrr(F)r.r/ pad_token_id padding_idx vocab_sizer EmbeddingrH embed_tokens ModuleListrangenum_hidden_layersr>layersr{r*rVr% rotary_embgradient_checkpointing post_initr-s r<r/AfmoeModel.__init__s  !.. ++LL):):FO   --i8M  CRC^==?de  << (;(;A(>}G[G[\_ooL'11!4L?-FF++!."0#2 K#5"C{"C%F%U%U#  &  ;; " ")[[-D-Dc-IJM"oomJ )$++ 6 A)2;;3J3J13MN).#$7 M!7 -0 %+/8O  >B  r>)rrrxrvrVrorwrp)NNNNNN)rlrmrnrorr#r/rrr!rJrNrprPr rOrrrtrrirvrwrxs@r<rkrk s { .2.22604(,!%< ##d*<  t+< ((4/ < &&- <  < $;< +,<  ' '<  < r>rkc\^\rSrSrSS0rSS0rSS/S/40rU4Sjr\\ SS \ RS -S \ RS -S \ RS -S \ S -S\ RS -S\ RS -S\S -S\S -S\\ R-S\\S\4Sjj55rSrU=r$)AfmoeForCausalLMigzlm_head.weightzmodel.embed_tokens.weightlm_headcolwise_gather_outputrlogitsc>[TU]U5 [U5UlURUl[ R "URURSS9UlUR5 gr) r.r/rkrSrprrrHrryrs r<r/AfmoeForCausalLM.__init__msS  '  ++yy!3!3V5F5FUS  r>Nr{r rdrTr|labelsrIoutput_router_logitslogits_to_keepr r@c UbUOURRnUR"SUUUUUUUS.U D6n U Rn [ U [ 5(a [ U *S5OU n URU SS2U SS245nSnUbUR"XUR40U D6n[UUU RU RU RU RS9$)ao Example: ```python >>> from transformers import AutoTokenizer, AfmoeForCausalLM >>> model = AfmoeForCausalLM.from_pretrained("meta-afmoe/Afmoe-2-7b-hf") >>> tokenizer = AutoTokenizer.from_pretrained("meta-afmoe/Afmoe-2-7b-hf") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N)r{r rdrTr|rIr)lossrrTrrUrrK)r(rrSrr\rsslicer loss_functionrprrTrrUr)r8r{r rdrTr|rrIrrr outputsr slice_indicesrrs r<riAfmoeForCausalLM.forwardtsB%9$D $++JjJj +/** + )%+'!5 +  +  11 8B>SV8W8W~ot4]k mA}a,?@A  %%fdooPPD(#33!//))!//   r>)rrSrp) NNNNNNNNr)rlrmrnro_tied_weights_keys_tp_plan_pp_planr/rrrJrNrpr rPrOrsrrrrirvrwrxs@r<rrgs(*,GH23H_-z:;H.2.204(,26*.!%,0-.< ##d*<  t+< &&- <  < ((4/ <   4'< $;< #Tk< ell*< +,<  #< < r>r)rrkrR)r")r2)Fcollections.abcrtypingrrJrrr[ activationsr cache_utilsr r generationr integrationsr r rr masking_utilsrrmodeling_layersrmodeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrrutils.output_capturingr r!configuration_afmoer#Moduler%r{rrrrrrrprsrrNrrr>rRrkr__all__rKr>r<rs*% &!.) S9QKF&bbGE,><299>(*+,2 UU\\ U# U%,, U&% II% <<% % << % LL4' %  %%'(%2)*V)RYYV)+V)r?2?D,,?,,^V %V V rJ +_J J Z Er>