Zj]SrSSKJr SSKrSSKJr SSKJr SSKJ r SSK J r SS K JrJr SS KJr SS KJr SS KJr SS KJrJr SSKJrJr SSKJr SSKJrJ r J!r!J"r" SSK#J$r$ SSK%J&r&J'r' SSK(J)r) SSK*J+r+J,r,J-r-J.r.J/r/J0r0J1r1 SSK2J3r3 \"Rh"\55r6"SS\-5r7"SS\.5r8"SS\Rr5r:"SS\Rr5r;"S S!\Rr5r<"S"S#\Rr5r="S$S%\Rr5r>"S&S'\)5r?\ "S(S)\,55r@\ "S*S+\+55rA"S,S-\@\5rB"S.S/\\@5rC/S0QrDg)1zPyTorch JetMoe model.)CallableN)nn) functional)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)create_causal_mask) GenericForSequenceClassification)MoeCausalLMOutputWithPastMoeModelOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)merge_with_config_defaults)OutputRecordercapture_outputs)LlamaDecoderLayer) MixtralModelMixtralPreTrainedModelMixtralRMSNormMixtralRotaryEmbeddingapply_rotary_pos_embeager_attention_forwardload_balancing_loss_func) JetMoeConfigc\rSrSrSrg) JetMoeRMSNorm4N__name__ __module__ __qualname____firstlineno____static_attributes__r(z/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/jetmoe/modular_jetmoe.pyr&r&4r/r&c\rSrSrSrg)JetMoeRotaryEmbedding8r(Nr)r(r/r0r3r38r1r/r3cB^\rSrSrS\S\S\SS4U4SjjrSrS rU=r$) JetMoeParallelExperts< num_experts input_size output_sizereturnNc>[TU]5 [R"[R "XU55UlXlX lX0l g)aS Initialize the JetMoeParallelExperts module. The experts weights are stored in [num_experts, output_size, input_size] format. Such that it's compatible with many MoE libraries, such as [Megablock](https://github.com/databricks/megablocks) and [ScatterMoE](https://github.com/shawntan/scattermoe), as well as the [MoE kernel](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/fused_moe/fused_moe.py) used in vllm. Args: num_experts (int): Number of experts. input_size (int): Size of the input. output_size (int): Size of the output. N) super__init__r Parametertorchemptyweightr8r9r:)selfr8r9r: __class__s r0r>JetMoeParallelExperts.__init__=s<" ll5;;{#TU &$&r/cURUSS9n/n[UR5H8nUR[R "X5UR U55 M: [R"USS9nU$)z Forward pass of the JetMoeParallelExperts module. Args: inputs (Tensor): Input tensor. expert_size: Expert size information. Returns: Tensor: Output tensor. rdim) splitranger8appendFlinearrBr@cat)rCinputs expert_size input_list output_listiresultss r0forwardJetMoeParallelExperts.forwardTsh\\+1\5  t''(A   qxx t{{1~F G)))KQ/r/)r9r8r:rB r*r+r,r-intr>rUr. __classcell__rDs@r0r6r6<s.'C'S's't'.r/r6c>^\rSrSrS\S\S\4U4SjjrSrSrU=r$)JetMoeTopKGatingir9r8top_kcz>[TU]5 X lXlX0l[ R "XSS9Ulg)z Initialize the top-k gating mechanism. Args: input_size (`int`): Size of the input. num_experts (`int`): Number of experts. top_k (`int`): Number of top experts to select. FbiasN)r=r>r8r9r^rLinearlayer)rCr9r8r^rDs r0r>JetMoeTopKGating.__init__js2 &$ YYzUC r/czURU5R5nURURSS9up4[R "USS9R U5n[R"URS5UR/URURS9nURSUS5nUR5RS5nUR5nUR!5n U R#S5upU R%URSS9n UR!5nX[n XXU4$)Nr#rGrdtypedevicetrunc) rounding_mode)rcfloattopkr^r@softmaxtype_aszerossizer8rgrhscatterlongsumtolistflattensortdiv)rC hidden_stateslogits top_k_logits top_k_indices top_k_gatesrogatesrP top_k_experts_index_sorted_experts batch_index batch_gatess r0rUJetMoeTopKGating.forward~s"M*002&,kk$**!k&D# mmLa8@@O    a $"2"2 3;;L;LU`UgUg  a2jjl&&q) "((* &--/ "/"4"4Q"7*..tzz.Q "))+ !7 #+FRRr/)r9rcr8r^rWrZs@r0r\r\is-D3DSDD(SSr/r\c:^\rSrSrSrS\4U4SjjrSrSrU=r $) JetMoeMoEz A Sparsely gated mixture of experts layer with 1-layer Feed-Forward networks as experts. Args: config: Configuration object with model hyperparameters. configc<>[TU]5 URUlURUl[ UR Ul[RR[R"UR55Ul [URURURS-5Ul[URURUR5Ul[#URURUR$S9Ulg)Nrr9r8r^)r=r> hidden_sizer9intermediate_sizeractivation_function activationr@rr?rArar6num_local_experts input_linear output_linearr\num_experts_per_tokrouterrCrrDs r0r>JetMoeMoE.__init__s  ,,!33 !;!;<HH&&u{{4??'CD 1&2J2JDOO]a]m]mpq]qr263K3KTM]M]_c_n_no&00,,  r/cUR5up#nURSU5nURU5upVpxn Xn URX5n U R SSS9n UR U S5U S-n UR X5n XSS2S4-n [R"X#-UR4U RU RS9nURSXm5nURX#UR5nXR-nU$)z Forward pass of the mixture of experts layer. Args: layer_input (Tensor): Input tensor. Returns: Tensor: Output tensor. Tensor: Router logits. rrGrr#Nrf)rpreshaperrchunkrrr@ror9rgrh index_addviewra)rC layer_inputbszlengthemb_sizerrrrP router_logits expert_inputsrxchunked_hidden_statesexpert_outputsro layer_outputs r0rUJetMoeMoE.forwards!, 0 0 2X!))"h7 BF++kBZ? -#0 ))-E - 3 3A2 3 >(=a(@ADYZ[D\\ ++MG'ag*>> S\4??;>CWCW`n`u`uvq+F #((dooF #ii/ r/)rrarrr9rr) r*r+r,r-__doc__r$r>rUr.rYrZs@r0rrs | r/rcF^\rSrSrSrS\4U4SjjrSrSrSr Sr U=r $) JetMoeMoAz A Sparsely gated mixture of attention layer with pairs of query- and output-projections as experts. Args: config: Configuration object with model hyperparameters. rcd>[TU]5 URUlURUlUR UR-UlURUl [RR[R"UR 55Ul[URUR UR5Ul[URURUR 5Ul[%UR URURS9Ulg)Nr)r=r>rr8rr9 kv_channelsnum_key_value_headsrr^r@rr?rArar6rrr\rrs r0r>JetMoeMoA.__init__s !33 ,,!--0J0JJ// HH&&u{{4??'CD 1$2B2BDOOUYUeUef243C3CTEUEUW[WfWfg&((**  r/cUR5up#nURSU5nURU5upVpxn XVXx4n Xn URX5n [R "X#-UR -UR4U RU RS9n U RSX\5nURX#UR S5nXU 4$)zq Map inputs to attention experts according to routing decision and compute query projection inside each experts. rrfr) rprrrr@ror^rrgrhrr)rCrrrrrrrrPr topo_inforrrors r0map JetMoeMoA.maps !, 0 0 2X!))"h7 UYU`U`alUmR;]) Q $0 **=F \DJJ &(8(8 9AUAU^l^s^s q*>O #((djj"E I55r/cUR5up4pVURSU5nUupxpXn URX5n XSS2S4-n [R"X4-UR 4U R U RS9n U RSX5nURX4UR 5nXR-nU$)ze Compute output projection inside each attention experts and merge the outputs of different experts. rNrfr) rprrr@ror9rgrhrrra)rCrrrrkrrrrrPrrrors r0reduceJetMoeMoA.reduces'2&6&6&8#Q!))"k: FOC;$9 ++MG(ag*>> S\4??;>CWCW`n`u`uvq+F #((dooF #ii/ r/c[S5e)Nz-This module doesn't support call and forward.)NotImplementedError)rCrs r0rUJetMoeMoA.forwards!"QRRr/)rarrr9r8rrr^) r*r+r,r-rr$r>rrrUr.rYrZs@r0rrs* | $6.,SSr/rc^\rSrSrSrSS\S\S-4U4SjjjrSS\RS\RS-S \RS-S \ S-S \ \R\RS-\ \RS-44 S jjr S rU=r$)JetMoeAttentioni z@ Multi-headed attention from 'Attention Is All You Need' paper. Nr layer_idxcZ>[TU]5 XlX lSUlUc-[ R SURRS35 SUl URUl URUl URUR-UlURUlUR UlURUlUR$S-Ul[)U5Ul[,R.R1UR2URS-SS 9Ulg) z Initialize the JetMoeAttention module. Args: config: Configuration object with model hyperparameters. layer_idx: Index of the layer in the model. TNzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.r#grFr`)r=r>rr is_causallogger warning_oncerDr*num_key_value_groupsrr^attention_dropoutrrkv_projection_sizenum_attention_heads num_headshead_dimscalingrexpertsr@rrbrkv_projrCrrrDs r0r>JetMoeAttention.__init__%s  "      !8!8 9:,,  %&!// !'!9!9"("4"4v7Q7Q"Q#)#=#= 33** }}d*  ( xxv'9'94;R;RUV;V]bc r/rxattention_maskposition_embeddingspast_key_valuesr;c URSSn/UQSPURP7nURRU5upn UR U5R SSS9upUR U5RSS5nU R U5RSS5n U R U5RSS5n Uup[XX5upUbURXUR5up[R"URR[5nU R!SUR"SS5n U R!SUR"SS5n U"UUU U U4UR$(dSO UR&UR(S.UD6unnUR "/UQUR"PSP76nURR+UU 5nUR "/UQSP76nUUU 4$)NrrrGr#)dropoutr)shaperrrrrr transposer updaterr get_interfacer_attn_implementationr!repeatr^trainingrrr)rCrxrrrkwargs input_shape hidden_shape query_statesrr key_states value_statescossinattention_interface attn_output attn_weightss r0rUJetMoeAttention.forwardFs$))#2. 88b8$--8 151A1A-1P. Y#'<< #>#D#DQB#D#O #((6@@AF __\2<r@Tensor LongTensorr tuplerUr.rYrZs@r0rr sd|dd ddH/37;(, /8||/8 t+/8#--4 /8  /8 u||U\\D0% 2E2LL M/8/8r/rc^\rSrSrSS\S\S-4U4SjjjrSS\RS\RS-S\RS-S \ S-S \ S-S \ \R\R4S-S \ \S \R4SjjrSrU=r$)JetMoeDecoderLayerixNrrc>[TU]X5 [UR5Ul[ X5Ul[UR5Ul[U5Ul U? gr) r=r>r&rinput_layernormrself_attentionpost_attention_layernormrmlp self_attnrs r0r>JetMoeDecoderLayer.__init__ysT +,V-?-?@-f@(5f6H6H(I%V$ Nr/rxr position_idsr use_cacherrr;c UnURU5nUR"SUUUUUUS.UD6un n X-nUnURU5nURU5nX-nU$)N)rxrrrrrr()rrrr) rCrxrrrrrrresidualrs r0rUJetMoeDecoderLayer.forwards!,,]; "11 ')%+ 3    q!!0 !55mD /  0 r/)rrrrr)NNNFN)r*r+r,r-r$rXr>r@rrr boolrrrrUr.rYrZs@r0rrxs|d /304(,!&HL|| t+&&-    $; #5<<#=>E+, r/rc\rSrSr%\"\SS9\"\SS9/\\"\SS9S.r\ \ S'Sr S r S /r S /rS rS rS r\R&"5S 5rSrg)JetMoePreTrainedModelir)indexr#)rrx attentionsrmodelFrrTc>[R"X5 [U[5(a5[R "UR SURRS9 g[U[[-5(a![R"UR5 gg)zInitialize the weights.r)meanstdN) r _init_weights isinstancer6initnormal_rBrinitializer_rangerrzeros_ra)rCmodules r0r#JetMoePreTrainedModel._init_weightsse %%d3 f3 4 4 LLSdkk6S6S T  I 5 6 6 KK $7r/r(N)r*r+r,r-rrr\r_can_record_outputsr$__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_can_compile_fullgraphr@no_gradrr.r(r/r0rrs)BNScklDmn+$_A> &+#-.#4"5N" ]]_%%r/rc^\rSrSrS\4U4Sjjr\\\SS\ RS-S\ RS-S\ RS-S\ S-S \ RS-S \S-S \\S \4S jj555rSrU=r$) JetMoeModelirc >[TU]U5 URUlURUl[ R "URURUR5Ul[ R"[UR5Vs/sHn[X5PM sn5Ul URUl[URUR S9Ulgs snf)N)eps)r=r> pad_token_id padding_idx vocab_sizer Embeddingr embed_tokens ModuleListrJnum_hidden_layersrlayersrr& rms_norm_epsnormrs r0r>JetMoeModel.__init__s  !.. ++LL):):FO   --i8M  CRC^==?de  << (;(;A(>}G[G[\_ooL'11!4L(;;')+%  & #oomJ![[)H4;;+H+HIM)$7* /#) MJ -0 %++  r/)rrrrrr)NNNNNN)r*r+r,r-r$r>rrrr@rrr FloatTensorrrrrrUr.rYrZs@r0rrs O| O .2.204(,26!%5 ##d*5  t+5 &&- 5  5 ((4/ 5 $;5 +,5  5  5 r/rc:^\rSrSrSS0rU4Sjr\\SS\RS-S\RS-S\RS-S \ S-S \RS-S \RS-S \ S-S \\R-S\ S-S\4Sjj55rSrU=r$)JetMoeForCausalLMizlm_head.weightzmodel.embed_tokens.weightcl>[TU]U5 [U5UlURUlUR Ul[ R"URURSS9Ul URUl URUl URUl UR5 g)NFr`)r=r>rrr aux_loss_coefrrbrlm_headtie_word_embeddingsrr8r post_initrs r0r>JetMoeForCausalLM.__init__s   (  ++#11yy!3!3V5F5FUS #)#=#= !33#)#=#=  r/Nr rrrr!labelsrlogits_to_keepoutput_router_logitsr;c \UR"SUUUUUUU S.U D6n U Rn [U[5(a [ U*S5OUn UR U SS2U SS245nSnUb*UR "UU4SURR0U D6nSnU (aY[U RURURU5nUb*XRURUR5-- n[!UUUU R"U R$U R&U RS9$)N)r rrrr!rr8r)lossaux_lossryrrxrrr()rr#rrXslicer2 loss_functionrrr"rr8rr1torhrrrxr)rCr rrrr!r6rr7r8routputsrx slice_indicesryr:r;s r0rUJetMoeForCausalLM.forwardsN+/** + )%+'!5 +  +  11 8B>SV8W8W~ot4]k mA}a,?@A  %% ;;11 D /%%  (( H !**X[[-EEE(#33!//))!//  r/)r1r2rr8rr3r) NNNNNNNrF)r*r+r,r-_tied_weights_keysr>rrr@rrr r-rrXrrUr.rYrZs@r0r/r/s*,GH .2.204(,26*.!%-.,19 ##d*9  t+9 &&- 9  9 ((4/ 9   4'9 $;9 ell*9 #Tk9  #9 9 r/r/c\rSrSrSrg)JetMoeForSequenceClassificationiRr(Nr)r(r/r0rDrDRs`cr/rD)r/rrrD)Ercollections.abcrr@rtorch.nnrrLrr activationsr cache_utilsr r generationr masking_utilsr modeling_layersr modeling_outputsrrmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrutils.output_capturingrrllama.modeling_llamarmixtral.modeling_mixtralrrrrr r!r"configuration_jetmoer$ get_loggerr*rr&r3Moduler6r\rrrrrrr/rD__all__r(r/r0rYsS$ $&!.)/RF&RR7E4/   H % N  2 *BII*Z.Sryy.Sb7 7tIS ISXU8biiU8p&*&R%2%%2E ,E E PK -K \d&FH]c kr/