Zj% %SSKJr SSKrSSKrSSKrSSKrSSKJr SSKJ r SSK J r J r SSK Jr SSKJr \"5(a*SSKrSSKJr SS KJr \RR+5r\ R."\5rSFSGS jjrSHS jrSISJS jjr\"5(a|\R:\R<\R>\R@\RB\RD\RF\RH\RJ\RL\RNS . r(SKSjr)Sr*SLSMSjjr+SNSOSjjr,Sr-"SS\R\R^5r0"SS\R\R^5r1"SS\R\R^5r2"SS\R\R^5r3"SS\R\R^5r4Sr5Sr6Sr7S r8S!r9SFSPS"jjr:"S#S$5r;"S%S&\;5r<"S'S(\;5r="S)S*\;5r>"S+S,\;5r?"S-S.\<5r@"S/S0\?5rA"S1S2\;5rB"S3S4\;5rC"S5S6\;5rD"S7S8\;5rE"S9S:\;5rF"S;S<\;5rG"S=S>\5rH\H"5rIS>\JS?'SQS@jrKSRSAjrLSBrMSCrNSSSDjrOSErPg)T) annotationsN)reduce)DistributedConfig)is_torch_greater_or_equallogging)GeneralInterface)is_torch_available)nncUbUc [S5eUbUb [S5eUGc[S5(d [S5e[RR 5R nUS:Xa [S5e[[U5n[RR5(d[[RS5n[[RS 5n[[RS 5nS S S SSS.n U RU5n [RRXUS9 [[U5nUS:waUR!U5 US:waTUR![[RS 55 UR%5n [R&"XL5n U nO![R&"U5n U=(d 0nUbUO[RR)5n[RR+U R U45nOUR,S:a SUR.;a [S5eUSnUR15n[R&"UR2S[[RS 535nX2U4$!["an [S5U eSn A ff=f)z Sets up the device mesh and initialized the backend for tensor parallelism. This function is called when the model is loaded and the TP plan is set to 'auto'. Nz-tp_plan has to be set when tp_size is passed.zY`tp_plan` and `device_map` are mutually exclusive. Choose either one for parallelization.z2.5z3Tensor parallel is only supported for `torch>=2.5`.mpsz3Tensor parallelism is not supported on MPS devices.RANK LOCAL_RANK WORLD_SIZEncclglooxcclhcclneuron)cudacpuxpuhpur)backendrank world_sizerzWe tried to initialize torch.distributed for you, but it failed. Make sure you init torch distributed in your script to use `tp_plan`.tpzsWhen using `tp_plan` and n-d `device_mesh`, it must contain a 'tp' dimension. Please provide a valid `device_mesh`.:) ValueErrorrOSErrortorch_C_get_acceleratortype RuntimeErrorgetattr distributedis_initializedintosenvirongetinit_process_group set_device Exceptioncurrent_devicedeviceget_world_sizeinit_device_meshndimmesh_dim_namessize device_type)tp_plantp_size device_mesh device_mapr8r1r local_rankr backend_mapreindex tp_devices z/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/integrations/tensor_parallel.pyinitialize_tensor_parallelismrC(s{wHIIz5tuu(//OP Phh//166 % TU U  4  //11 2::f-. L!9:  L!9: '-fVTZfno %//+6!!44W\f4g!( !<%'"--j9 %   % %c"**\*B&C D"113E [8I"J [1I$*J$0'e6G6G6V6V6X''88'T   a ;555 <&d+K""$\\[%<%<$=Qs2::lC[?\>]"^_ G ++9 W s/B2J.. K 8 KK c4[R"SSU5$)aS Replace the numbers in the `name` by wildcards, only if they are in-between dots (`.`) or if they are between a dot (`.`) and the end of the string. This matches how modules are named/numbered when using a nn.ModuleList or nn.Sequential, but will NOT match numbers in a parameter name itself, e.g. if the param is named `"w1"` or `"w2"`. z \.\d+(\.|$)c*SURS5-$)Nz.*rgroup)ms rB2replace_layer_number_by_wildcard..psD1771:,=)resub)names rB replace_layer_number_by_wildcardrOis 66."=t DDrKc[U5nX1;aX$U(a$SU;aURSS5S=oA;aX$g)a Get the TP style for a parameter from the TP plan. The TP plan is a dictionary that maps parameter names to TP styles. The parameter name can be a generic name with wildcards (e.g. "*.weight") or a specific name (e.g. "layer_1.weight"). The `is_weight` is important because for weights, we want to support `.weights` and `.bias` cases seamlessly! but not parent classes for `post_init` calls .rrN)rOrsplit)parameter_namer9 is_weightgeneric_param_name module_names rB_get_parameter_tp_planrWssU:.I$** s00EWE^E^_bdeEfghEi6ik5u## rK) BOOLU8I8I16F16BF16I32F32F64I64F8_E4M3c[U[5(a9[U5nX-S:Xd SUSU35eX-nUVs/sHoCU-PM sn$X-S:Xd SU35eX-nU/U-$s snf)a Convert block count or proportions to block sizes. This function accepts - The number of blocks (int), in which case the block size is total_size//blocks; or - A list of block sizes (list[int]). In the second case, if sum(blocks) < total_size, the ratios between the block sizes will be preserved. For instance, if blocks is [2, 1, 1] and total_size is 1024, the returned block sizes are [512, 256, 256]. rz Cannot split z in proportional blocks: zPrepacked is not divisible by ) isinstancelistsum) total_sizeblocks total_blocks part_sizeblock single_sizes rB_blocks_to_block_sizesrms&$6{ (A-lzlJcdjck/ll-. /56veE!v66"a'R+I&)RR' * }v%% 7sA+cUnURUnUR5n[USS9n/n Sn UH*n X-n X<-n US-U -nU [X-X-5- n X- n M, UR 5nSnUS:XdUS:Xa$USR [ R5nS nUS:XaXYS4nO:US:XdUS :Xa US S 2U S4nO#US:XdUS :XaUSU 4nO[S US35eU(aU$UR [U5$)u When weights are packed (gate_up_proj), we need to make sure each shard gets its correct share. So if you have: gate_proj ( 16, 5120, 8190) and up_proj ( 16, 5120, 8190) packed as gate_up_proj ( 16, 5120, 2 * 8190) And you shard along the last dimension, you need to interleave the gate and up values: Now, if we shard along the last dimension across TP_size (Tensor Parallelism size), we must interleave the values from gate and up projections correctly. Let's take TP_size = 4 for an example: Packed tensor `gate_up_proj` --------------------------------------------------------------- [ G0 G1 G2 G3 | G4 G5 G6 G7 | ... | U0 U1 U2 U3 | U4 U5 U6 U7 | ... ] ↑─────────────↑ ↑─────────────↑ ↑─────────────↑ ↑─────────────↑ Gate Slice 0 Gate Slice 1 Up Slice 0 Up Slice 1 Explanation: - The first half of the tensor (left of the center) holds the gate_proj values. - The second half (right of the center) holds the up_proj values. - For TP=4, we divide each half into 4 slices. In this example, we show two slices for brevity. - Each shard receives one slice from the gate part and the corresponding slice from the up part. For instance: • Shard 0 gets: [ Gate Slice 0, Up Slice 0 ] = [ G0, G1, G2, G3, U0, U1, U2, U3 ] • Shard 1 gets: [ Gate Slice 1, Up Slice 1 ] = [ G4, G5, G6, G7, U4, U5, U6, U7 ] • … and so on. This ensures that each shard receives an equal portion of both gate and up projections, maintaining consistency across tensor parallelism. r)rgrhrrFrbF8_E5M2.TNzUnsupported dim z", only dim 0, 1 or 2 are supported) shaper7rmrange get_dtypetor"float16r str_to_dtype)param empty_paramr;rdimslice_rgr block_sizestensors_slices block_offset block_sizeshard_block_sizestartstop slice_dtypecastedtensors rBget_packed_weightsrsG>F""3'J!!#J(JqIKNL! %3'q,,% 4l6IJJ" """$KFi;)#; . ax+, SBY>3./ SBY^+,+C50RSTT  yyk233rKcUS:wa [S5eUS:aUO XR-nURUnXS-nXb-nURSUnURUS-Sn UR"/UQUPUPUPU Q76n [ U5n [ U5S-n [ [ U R55n XXsX'X'U R"U 6nURU5nU$)a? Reorders a tensor that was reconstructed from sharded packed weights into its canonical packed format. For example, if a weight was packed (e.g., gate_proj and up_proj) and then sharded, DTensor.full_tensor() might produce an interleaved layout like [G0, U0, G1, U1, ...] along the sharded dimension. This function reorders it to [G0, G1, ..., U0, U1, ...]. This is an inverse operation to get_packed_weights. Args: reconstructed_tensor: The tensor reconstructed from DTensor (e.g., via .full_tensor().contiguous()). sharded_dim: The dimension index in the reconstructed_tensor that was originally sharded. world_size: The tensor parallel world size. num_packed_projs: The number of projections that were packed together (e.g., 2 for gate_up_proj). Returns: The reordered tensor in canonical packed format. rzNum blocks different from 2 is not supported yet. This is most likely a bug in your implementation as we only pack gate and up projections together.rNr) r r5rrviewlenrerspermute reshape_as)packed_parameter sharded_dimr num_blocksactual_sharded_dimtotal_size_on_sharded_dimoriginal_block_size_on_dimshard_chunk_size prefix_shape suffix_shape tensor_view axis_ws_abs axis_npp_abs permute_ordertensor_permutedfinal_ordered_tensors rBrepack_weightsrs:0Q c  )4q(8kLaLa>a 0 6 67I J!:!H1?#))*=+=>L#))*K>Y[h[u;M ;!))=9O+556FG rKcdURnURn[[RU5n[ U[ R5(a[UR5OUR5n US:aXd-nUR5S:XaUS:Xa[U 5S:XaSnO+UR5S:XaUS:Xa[U 5S:XaSn[R"XU- 5n X:-n [X-X5n XF:a[SUSU35eX8:a[SUSU35eUbaUR5S:XaMUS:XaG[U 5S:Xa8Xs=::aU :aO OUS S $[ R "/[ R"US 9$[%S 5/[U 5-n XU:aF[%X5X'U['U 5n[ U[5(aUVs/sHoS S PM nnU$SX'[ R "['U 5[ R"S 9$s snf) a} Generalized tensor sharding across a multi-dimensional device mesh. Extract only the fraction of the parameter owned by the given `rank` when the parameter would have gone sharding at provided `dim`. Extraction follows the pytorch `Shard` placement so that sharding and materializing back to full tensor follows `Shard` semantics. `Shard` follows torch.chunk style sharding of the tensor. We demonstrate some cases below on how sharding happens including some edge cases such as some ranks having an empty tensor as shard. Below implementation is robut to all these cases. Case (1) empty_param (16, 5120, 8190) dim 0 device_mesh.size() 4 rank 0 gets (4, 5120, 8190) (0 ... 4, 5120, 8190) rank 1 gets (4, 5120, 8190) (4 ... 8, 5120, 8190) rank 2 gets (4, 5120, 8190) (8 ... 12, 5120, 8190) rank 3 gets (4, 5120, 8190) (12 ... 16, 5120, 8190) Case (2) empty_param (16, 5120, 8190) dim 0 device_mesh.size() 14 rank 0 gets (2, 5120, 8190) (0 ... 2, 5120, 8190) rank 1 gets (2, 5120, 8190) (2 ... 4, 5120, 8190) rank 2 gets (2, 5120, 8190) (4 ... 6, 5120, 8190) rank 3 gets (2, 5120, 8190) (6 ... 8, 5120, 8190) rank 4 gets (2, 5120, 8190) (8 ... 10, 5120, 8190) rank 5 gets (2, 5120, 8190) (10 ... 12, 5120, 8190) rank 6 gets (2, 5120, 8190) (12 ... 14, 5120, 8190) rank 7 gets (2, 5120, 8190) (14 ... 16, 5120, 8190) rank 8 gets (0, 5120, 8190) rank 9 gets (0, 5120, 8190) rank 10 gets (0, 5120, 8190) rank 11 gets (0, 5120, 8190) rank 12 gets (0, 5120, 8190) rank 13 gets (0, 5120, 8190) Case (3) empty_param (16, 5120, 8190) dim 0 device_mesh.size() 3 rank 0 gets (6, 5120, 8190) (0 ... 6, 5120, 8190) rank 1 gets (6, 5120, 8190) (6 ... 12, 5120, 8190) rank 2 gets (4, 5120, 8190) (12 ... 16, 5120, 8190) In case (2), empty shards are returned with appropriate dimension to allow for operations to work smoothly. Args: param (torch.Tensor): The tensor to shard. empty_param (torch.Tensor): A tensor used for shape reference. device_mesh (torch.Tensor): Shape [d_0, ..., d_n] representing the mesh. rank (int): Global rank of the current process/device. dim (int): Dimension along which to shard the tensor. rrrzdim z* is out of bounds for tensor of dimension zRank z is out of bounds for mesh size Ndtyper2)r)r5rrroperatormulrdr"Tensorre get_shaperzrmathceilminr emptyint64slicetuple)rxryr;rrz tensor_idx param_dim mesh_shaper param_shape shard_sizerend slice_indicesps rBget_tensor_shardr9sh  I""J j1J'1%'F'F$u{{#EOOL]K QwoA#(s;/?1/D  a C1H[1AQ1F;+j89J  E e +"2 3C 4u$NykZ[[ 5&FzlSTT+//"3q"8SAX#kJZ^_J_  $ $8O;;rTB B4[MC $44M 3"5. eM*+ eT " "#()5aqT5E) K ;;u[) == *s+H-c,[R"XSS9$)z9Split tensor along last dimension into world_size chunks.rqrz)r"chunk)xrs rB_split_along_last_dimrs ;;q" --rKc8\rSrSrSr\S5r\S5rSrg)_AllReduceBackwardizRIdentity forward, all-reduce backward. Used before colwise layers (f in Megatron).cX lU$N)r;ctxrr;s rBforward_AllReduceBackward.forwards %rKcURnUR5S:XaUS4$UR5n[R"U[R R UR5S9 US4$NroprG)r;r7 contiguousdist all_reduceReduceOpSUM get_group)r grad_outputr;s rBbackward_AllReduceBackward.backwardscoo     "$ $!,,.    (9(9AVAVAXYD  rKN __name__ __module__ __qualname____firstlineno____doc__ staticmethodrr__static_attributes__rrKrBrrs+\!!rKrc8\rSrSrSr\S5r\S5rSrg)_AllReduceForwardizQAll-reduce forward, identity backward. Used after rowwise layers (g in Megatron).cUR5S:XaU$[R"U[RRUR 5S9 U$r)r7rrrrrrs rBr_AllReduceForward.forwards@     "H dmm//{7L7L7NOrKc US4$rr)rrs rBr_AllReduceForward.backwards D  rKrNrrrKrBrrs+[ !!rKrc8\rSrSrSr\S5r\S5rSrg) _AllGatheriz>@@Gs/ C cURnUR5nUS:XaUS4$UR5n[X5nXTR 5S4$Nrr;r7rrr)rrr;rrchunkss rBr_AllGather.backwardsYoo  %%' ?$ $))+&{?|&&($..rKrNrrrKrBrrs-FAA" / /rKrc8\rSrSrSr\S5r\S5rSrg)_Splitiz>Split forward, all-gather backward. Scatters replicated input.cX lUR5nUS:XaU$UR5n[X5nXTR 5$rr)rrr;rrrs rBr_Split.forwardsJ% %%' ?H))+&q5|&&((rKcURnUR5nUS:XaUS4$UR5S- nUR5nUR 5nUR 5n[ U5Vs/sHn[R"U5PM nnXU'[R"XUS9 [R"XS9R 5S4$s snfrr rrr;rrrrGrrs rBr_Split.backward oo  %%' ?$ $??$q())+%%'!,,. >CJ>OP>Ou'' 4>O P'D  >yy3>>@$FFQ7 CrNrrrKrBrrs-H ) )GGrKrc8\rSrSrSr\S5r\S5rSrg)_ReduceScatteri zCReduce-scatter forward, all-gather backward. For sequence parallel.cX lUR5nUS:XaU$UR5S- nUR5n[ UR X4S95n[ UR 5nXt==U-ss'[R"XqRURS9n[R"X[RRUS9 U$)Nrrrr)r;r7rzrrerrrr"rrr2rreduce_scatterrr) rrr;rrrG input_chunks output_shapeoutputs rBr_ReduceScatter.forward#s% %%' ?H557Q;%%'AGGJG=> AGG} :-\J FT]]5F5FeT rKcURnUR5nUS:XaUS4$UR5S- nUR5nUR 5nUR 5n[ U5Vs/sHn[R"U5PM nnXU'[R"XUS9 [R"XS9R 5S4$s snfrrrs rBr_ReduceScatter.backward6rrrNrrrKrBrr s-M$GGrKrc,[RX5$)zAIdentity forward, all-reduce backward. Use before colwise layers.)rapplyrr;s rBall_reduce_backwardrNs  # #A 33rKc,[RX5$)z@All-reduce forward, identity backward. Use after rowwise layers.)rrrs rBall_reduce_forwardrSs  " "1 22rKc,[RX5$)z#All-gather forward, split backward.)rrrs rBrrXs   A ++rKc,[RX5$)z#Split forward, all-gather backward.)rrrs rBsplitr ]s << ''rKc,[RX5$)z,Reduce-scatter forward, all-gather backward.)rrrs rBrrbs    //rKcp^^^TbURUU4Sj5 TbURUU4Sj5 U$)z Copy pasted from torch's function but we remove the communications (partitioning) as well as buffer registering that is similarly not efficient. c>T"XT5$rr)modinputsr;input_fns rBrI#distribute_module..rs XcS^=_rKc>T"XT5$rr)rroutputsr; output_fns rBrIrts )CZeBfrK)register_forward_pre_hookregister_forward_hook)moduler;rrs ```rBdistribute_modulergs4(()_`$$%fg MrKcr\rSrSrSrSrSrSrS SjrSr Sr S S Sjjr SSjr SS jr SS jrS rg)TensorParallelLayerixz.General tensor parallel layer for transformersNc(X lXlX0lgr)rr;ry)selfr;rrys rB__init__TensorParallelLayer.__init__s &&rKc[erNotImplementedErrorrrrr;s rB_prepare_input_fn%TensorParallelLayer._prepare_input_fn!!rKc[err rrrr;s rB_prepare_output_fn&TensorParallelLayer._prepare_output_fnr%rKc[err rrxrr2rs rB shard_tensor TensorParallelLayer.shard_tensors "!rKc H[UUURUR5 gr)rr#r(rrr;kwargss rBprepare_module_tp%TensorParallelLayer.prepare_module_tps"    " "  # #  rKc[U5$)z Compute the expected shape after TP sharding for a given full shape. Args: full_shape: The full (unsharded) parameter shape Returns: The expected sharded shape for this rank )r)r full_shapes rBget_expected_sharded_shape.TensorParallelLayer.get_expected_sharded_shapesZ  rKcg)z Update module attributes (e.g. in_features, out_features) to reflect sharded dimensions. Args: module: The module to update Returns: None, update the module in-place Nrrrs rBupdate_module_attributes,TensorParallelLayer.update_module_attributess rK)r;ryrNNNrx torch.Tensorr int | Nonereturnr=r nn.Moduler?rAr4ztuple[int, ...] | torch.Sizer?ztuple[int, ...]rrA)rrrrrr;rryrr#r(r,r1r5r9rrrKrBrrxsV8K DK' ""VZ"!"/9" "   !  rKrcn^\rSrSrSrS S U4SjjjrSrSrS S SjjrSSjr SSjr S r U=r $)ColwiseParalleliz Column-wise parallel: weight is sharded on dim -2 (output features). Forward: input replicated -> output sharded on last dim. If gather_output=True, output is all-gathered to produce full tensor. c 2>[TU]"S0UD6 XlgNr)superr gather_output)rrIr0 __class__s rBrColwiseParallel.__init__s "6"*rKc4U(aUSOUn[XC5$Nrrrrrr; input_tensors rBr#!ColwiseParallel._prepare_input_fns$*vay "<==rKc>UR(a [X#5$U$r)rIrr's rBr("ColwiseParallel._prepare_output_fns   g3 3rKcn[U[R5(aUR5O[ UR 55nUS:Xa-[ XRURURS5nO,[ XRURURS5nURX4S9$Nrrqrpr2r rdr"rrzrrrryr;rrurrxrr2rrz parameters rBr,ColwiseParallel.shard_tensors(u||< VZ 8! 8/9 8  8 ]]rKrEc4\rSrSrSrSrSrS SjrSrSr g) ReplicatedWithGradAllReduceia5 Replicated parameter with gradient all-reduce. For parameters like q_norm/k_norm that sit between colwise and rowwise layers. The parameter is replicated (not sharded), but its gradient accumulates from local heads only in TP mode. This class registers a backward hook to all-reduce the parameter gradient. cU$rrr"s rBr#-ReplicatedWithGradAllReduce._prepare_input_fns rKcU$rrr's rBr(.ReplicatedWithGradAllReduce._prepare_output_fnsrKNc&USRX4S9$N.rVrur+s rBr,(ReplicatedWithGradAllReduce.shard_tensorSz}}F}88rKc 2U4SjnURU5 g)Nc|UR5H(nURcM[URU5 M* gr) parametersgradr)r grad_inputrmeshrxs rB_backward_hookEReplicatedWithGradAllReduce.prepare_module_tp.._backward_hooks+)::)&uzz48*rK)register_full_backward_hook)rrr;r0rys rBr1-ReplicatedWithGradAllReduce.prepare_module_tps?J 9 **>:rKrr;) rrrrrr#r(r,r1rrrKrBriris9;rKric2\rSrSrSrSrSSjrS SjrSrg) MlaKvAProjParallelia For MLA attention used in DeepSeek-V2 style models (deepseek_v2, longcat_flash, glm_moe_dsa, glm4_moe_lite): kv_a_proj_with_mqa output is [kv_lora_rank + qk_rope_head_dim] (can have different naming but important thing to understand is that it is split) Example below (from modeling_longcat_flash.py): kv_a_proj_with_mqa | split / k_pass k_rot <-- "bypasses kv_b_proj" | | (goes straight to attention, kv_a_layernorm | never touches kv_b_proj) | | kv_b_proj | (colwise) | | | k_pass k_rot \ / cat | key_states k_pass is passed to kv_b_proj (colwise) which has built-in all_reduce_backward so we don't have a partial gradient for it. However, k_rot goes straight to attention, never touches kv_b_proj. So we need to average gradient across all ranks otherwise we only get gradient for one rank (partial gradient). c2[URS5(d"[S[U5RS35eURR nUR URSU- U/SS9upV[Xc5n[R"XV/SS9$)Nqk_rope_head_dimz Config for z does not have `qk_rope_head_dim`. MlaKvAProjParallel requires `qk_rope_head_dim` to be defined in the model config. Please add it to the model's config or update the TP plan mapping.rqr) r`configAttributeErrorr%rrr rrrr"r)rrrr;rope_dim pass_output rope_outputs rBr(%MlaKvAProjParallel._prepare_output_fnsszz#566 d3i0012UU  ::..#)<<b1AH1Lh0W]_<#` )+C yy+3< output partial -> all-reduce to replicate. Args: split_input: If True, splits replicated input before matmul. Use when input comes from a non-parallelizable operation (chunk/slice). Default False (expects pre-sharded input from colwise layer). c 2>[TU]"S0UD6 XlgrG)rHr split_input)rrr0rJs rBrRowwiseParallel.__init__<s "6"&rKc[US5(a%URbURUlSUlU(aUSOUnUR(a [ XC5$U$)Nbiasr)r`r_biasrr rOs rBr#!RowwiseParallel._prepare_input_fn@sQ 3  CHH$8CICH$*vay   3 3rKct[X#5n[US5(aURbX!R-nU$)Nr)rr`rr's rBr("RowwiseParallel._prepare_output_fnLs3$W: 3 SYY%: )GrKc [U[R5(aUR5O[ UR 55nUS:XaUSnO,[ XRURURS5nURX4S9$)Nr.rqrVrWrXs rBr,RowwiseParallel.shard_tensorRsr(u||< 6= ) )**+E!%!@!@!G!JF  *rKrrc)rrdr;r<rBrCrergs@rBrr1sQ'' VZ 8! 8/9 8  8 KKrKrc2\rSrSrSrSSSjjrSrg)PackedColwiseParallelisz@Packed column-wise parallel for fused weights like gate_up_proj.Nc0[U[R5(aUR5O[ UR 55nUS:Xa-[ XRURURS5nOURURR5nU[ U5:a-[ XRURURS5nO,[XRURURS5nURX4S9$rU)rdr"rrzrrrryr;rr5rrrru)rrxrr2rrzrYexpected_shapes rBr,"PackedColwiseParallel.shard_tensorvs(u||<[TU]"S0UD6 XlgrG)rHrr)rrr0rJs rBrEmbeddingParallel.__init__s "6"&<#rKcU(aUSOUnURS:XaYUR5nURRSnXV-nXv-nXG:XH:-n XlUR 5U- n SX'U $U$rM)rrweightrr _input_maskclone) rrrr;rPrper_partition_sizevocab_start_indexvocab_end_index input_mask masked_inputs rBr##EmbeddingParallel._prepare_input_fns$*vay   & &! +--/D "%!1!1!!4  $ 9 /DO':|?^_J(O(--/2CCL'(L $ rKcURS:Xa][US5(aLURnURS5R U5nX%)R UR 5-nU?[X#5$)Nrrrq)rr`r unsqueeze expand_asrurr)rrrr;r mask_expandeds rBr($EmbeddingParallel._prepare_output_fnsi  & &! +]0K0KJ&004>>wGM 3 3GMM BBG!'77rKc[U[R5(aUR5O[ UR 55nUS:Xa-[ XRURURS5nO7[ UURURURUR5nURX4S9$)NrrqrV) rdr"rrzrrrryr;rrrurXs rBr,EmbeddingParallel.shard_tensors(u||<[TU]"S0UD6 XlgrG)rHr sequence_dim)rruse_local_output use_dtensorr0rJs rBrSequenceParallel.__init__s "6"(rKc4U(aUSOUn[XC5$rM)rrOs rBr#"SequenceParallel._prepare_input_fns$*vay ,44rKc[X#5$r)rr's rBr(#SequenceParallel._prepare_output_fns g33rKc&USRX4S9$rorpr+s rBr,SequenceParallel.shard_tensorSz}}F}88rK)r)rFF)rr*rrdr;r< rrrrrrr#r(r,rrfrgs@rBrrsE ))5 4VZ9!9/99 99rKrcZ^\rSrSrSrU4SjrSS SjjrS SjrS SjrSr U=r $) GroupedGemmParalleli zZ Applies Expert Parallelism to MoE experts by loading the correct experts on each device. c &>[TU]"S0UD6 grGrHrrr0rJs rBrGroupedGemmParallel.__init__ "6"rKcvURRSnXPRR5-S:wa*[ SUSURR5S35eXPRR5-nUnUR U-nUR S-U-n [ U[R5(dUR5O URn Ub!Xs=::aU :aO OUSSRUS9$UcXU RX4S9$[U 5S:aUbgUSSRX4S9$)NrzAGlobal number of experts must be divisible by number of devices:  %  != 0r)r2rV) ryrrr;r7r rrdr"rrrur) rrxrr2rglobal_num_expertslocal_num_expertsrrrrrs rBr, GroupedGemmParallel.shard_tensorsM"--33A6  0 0 5 5 7 71 <STfSggjkok{k{lAlAlCkDDIJ /2B2B2G2G2II&  J&yy1} *)3E5<<)H)H!ekk  !e&?C&?8;;f;- -  s#&&f&B B Z1_!78;;f;: :rKczURR5n[U5nUSU-nXCS'[U5$rM)r;r7rer)rr4rrrrs rBr5.GroupedGemmParallel.get_expected_sharded_shape-s@%%**, Z !!H 2$aU|rKc[US5(a2URURRS45SUlgg)N num_expertsr)r`r5ryrrrr8s rBr9,GroupedGemmParallel.update_module_attributes5sB 6= ) )!%!@!@$BRBRBXBXYZB[A]!^_`!aF  *rKrr;r<rBrC) rrrrrrr,r5r9rrfrgs@rBrr sB#VZ;!;/9; ;0bbrKrcR^\rSrSrSrU4SjrSrSrSS SjjrSr U=r $) RouterParalleli:zI Allows to reshape the router scores to support running expert parallel. c &>[TU]"S0UD6 grGrrs rBrRouterParallel.__init__?rrKcU(aUS$U$rMrr"s rBr# RouterParallel._prepare_input_fnBs"vay..rKc2UR5UR5pT[USS5nUc[[USS5SS5nUc"[S[ U5R S35eXe-S:wa[ SUSUS 35eXe-nUupn X-U:gn U RU S 5n U RU S 5n US :a[R"X5n O(U RU S:S5RU S:S 5n U RU S :HU5n XU 4$) u Remap global expert indices to local and zero out non-local scores. Example: 4 tokens, top_k=4, 128 experts, EP=8. num_local_experts = 128/8 = 16. Router produces (all ranks see the same values): router_scores: (4, 4) — top-k routing weights router_indices: (4, 4) — global expert IDs [ 52, 42, 119, 67], [102, 89, 61, 40], [ 82, 103, 4, 34], [ 93, 23, 109, 11], Each index maps to a rank: index // 16 gives the owning rank. [3, 2, 7, 4], [6, 5, 3, 2], [5, 6, 0, 2], [5, 1, 6, 0], For rank 0 (owns experts 0-15), we remap local indices with fmod and fill non-local with sentinel=16 (used for one_hot masking): router_indices (rank 0): [ 16, 16, 16, 16], [ 16, 16, 16, 16], [ 16, 16, 4, 16], [ 16, 16, 16, 11], Scores for non-local experts are zeroed out via masked_fill: router_scores (rank 0): [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.3, 0.0], ← only expert 4 (local) keeps its score [0.0, 0.0, 0.0, 0.1], ← only expert 11 (local) keeps its score both router_scores and router_indices stay (seq, top_k) shape. They are paired element-wise: scores[i] is the weight for indices[i]. All expert forward implementations (grouped_mm, batched_mm, eager) flatten both with reshape(-1) and rely on this pairing. Changing the shape of one without the other breaks routing! Each rank believes it is alone and computes only its part of the hidden states. The sentinel index (num_local_experts) is skipped by one_hot encoding or clamped + masked in grouped_mm/batched_mm. After the expert forward, an all_reduce sums partial outputs across EP ranks to produce the full result. rNrzRouter module z. is missing num_experts and config.num_expertsrz>The number of experts must be divisible by number of ep_size: rrgrqr) rr7r'rr%rr masked_fillr"fmod) rrrr;ep_rankep_sizernum_local_experts router_logits router_scoresrouter_indicesnon_local_masks rBr(!RouterParallel._prepare_output_fnEsJ\'5579I9I9Kc=$7  !'#x"> tTK   >$s)2D2D1EEs!tu u  A %PQ\P]]`ah`iino (27>4 n(='I%11.#F '33NBG q "ZZJN+778JANZZ[ilm[moqrN'33Nb4HJ[\^;;rKc&USRX4S9$rorpr+s rBr,RouterParallel.shard_tensorrrKrr;r<rrgs@rBrr:sB#/D[TU]"S0UD6 grGrrs rBr!MoeTensorParalellExperts.__init__rrKcTUSnUSnUSn[XC5n[Xc5nXEU4$)NrrrrN)rrrr; hidden_states top_k_index top_k_weightss rBr#*MoeTensorParalellExperts._prepare_input_fns@q Qi q ,MG ,MG M::rKc[X#5$r)rr's rBr(+MoeTensorParalellExperts._prepare_output_fns !'77rKc&USRX4S9$rorpr+s rBr,%MoeTensorParalellExperts.shard_tensors Sz}}F}88rKrr;r<rrgs@rBrrs@#; 8 VZ9!9/99 99rKrc.\rSrSrSrSrSSjrSrSrg) MoeIdentityExpertParallelia5 TP class for zero/identity experts in MoE layers. Under TP, the parent MoeTensorParalellExperts does all_reduce_forward (sum) on the expert module output. Identity experts produce the same output on every rank, so the sum gives world_size * output. This class divides the input by world_size to compensate. cBU(aUSOUnXCR5- $rM)r7rOs rBr#+MoeIdentityExpertParallel._prepare_input_fns!$*vay ..000rKNc&USRX4S9$rorpr+s rBr,&MoeIdentityExpertParallel.shard_tensorrrrKc ,[XURS9 g)N)r)rr#r/s rBr1+MoeIdentityExpertParallel.prepare_module_tps&8N8NOrKrr;) rrrrrr#r,r1rrrKrBr r s1 9PrKr cf\rSrSr%\"5(a`\(aY\"SS9\"SS9\"SS9\"5\"5\"SS9\ "5\ "5\ "5\ "5\ "5\"5\"5\"5\"5S.O0rS S S S S S SSS S S S . rS \S'S S S S S S S S S S S S . rS \S'\SSj5r\SSj5rSrg )ParallelInterfaceirrrTrbr)embedding_rowwiseembedding_colwisecolwise_gather_outputcolwiserowwiserowwise_split_inputpacked_colwisepacked_rowwisesequence_parallel grouped_gemm ep_routermoe_tp_expertsmoe_identity_expertreplicated_with_grad_allreduce mla_kv_a_projrprqN) rrrrrrrrrr r!zdict[str, int | None]plan_to_weight_dimplan_to_bias_dimc X RU'gr)r"clskeyvalues rBregister_plan_to_weight_dim-ParallelInterface.register_plan_to_weight_dims&+s#rKc X RU'gr)r#r%s rBregister_plan_to_bias_dim+ParallelInterface.register_plan_to_bias_dim s$)S!rKr)r'strr(r>)rrrrr _torch_distributed_availablerrErrrrrrrr rir~_global_mappingr"__annotations__r# classmethodr)r,rrrKrBrrs*   $@!"3!!L!2!!L%44%H&(&(#2t#D3535!1!3/1')68#<#>.I.K/1 $'4!#!!*. 1-  !##!!!*. /+ ,,**rKrALL_PARALLEL_STYLESczUR5nSUR=(d 0;aURS5OSnUS:aURU-n[ U5Vs/sHn[ R "U5PM nn[R"X`R5US9 [ R"XaS9$s snf)a^ All-gather a sharded tensor along the specified dimension to reconstruct the full tensor. Args: local_tensor: The local shard of the tensor on this rank shard_dim: The dimension along which the tensor was sharded device_mesh: The device mesh for distributed communication Returns: The full reconstructed tensor (same on all ranks) rNrrFr) r7r6rr5rsr"rrrrr) local_tensor shard_dimr;r process_grouprgathered_tensorss rBgather_full_tensorr9s!!#J37KTR5ST3$)Nz TP Plan: )__repr__)current_module_planrsrBrI5add_tensor_parallel_hooks_to_module..sV__%6$7{CVBW"XrK) r3r1rr!loggerwarning _hf_tp_plan_hf_device_meshrK)modelrrL layer_namer;tp_layerr?s `` rB#add_tensor_parallel_hooks_to_modulerU}s,&&':;   & &v{<< & P1!,X'#  NN$ZL0`ag`hi   sA A8A33A8c XSU;aURSS5OUupUR=(d 0n URU5n [U5n[ X:5n [ R "5S:Xa8U c[RSUS35 O[RSUSU 35 Sn U bE[U n X-l X}l Xml U RUSXFS9nU(aUR5nOUSSR#U5n[%U[&R(R*5(d+[&R(R+XR-5S 9n[/XU5 U bU R1U 5 U$![a!n[!S US U S U S U35 SnANSnAff=f)a This function is called in `from_pretrained` when loading a model's checkpoints. It receives the pointer to the parameter (or the parameter itself) and takes care of "sharding". All process run this function, so they just load the partition of the tensor that they require. Main uses cases: - column / rowise parallelism, you just shard all the weights of the layer (weight and bias) - packed layers: you slice the weights, then shard like above - custom operation: - you want to add an all-gather at the end of a local layer. - you want to have a layer that is isolated from the rest of the world (because torch.DTensor does not work well with `.view` for instance) rQrrNzTensor sharding plan for z+ not found, using default 'replicate' plan.z: )rrr2rHrIz" Fix it's TP plan, current layer: z : ) requires_grad)rRr9 get_submoduler*rWrget_rankrNinfor3ryr;rr,rr!printrurdr"r Parameteris_floating_pointsetattrr9)rRrxryrSparam_casting_dtype is_contiguousrr;r@rAr9 module_to_tpcurrent_shard_planrTr?s rBshard_and_distribute_modulercs ?B^>S^223:YgJmm!rG&&z2L t9D/H }}!  % KK3JH#. #.  M))%DH[)iE((* a /0 eUXX// 0 0""58U8U8W"X Le,)),7 L#  $^$44deqdrsUV^U__bcdbef   s0AE>> F)F$$F)cUcgUVs1sHn[U5iM nn[U5nUR5nUHnSU;aURSS5SOUn[R "SSU5nXq;a%UR US5 URU5 MbSU;dMjURSS5S=o;dMUR US5 URU5 M [U5S:a[RSU35 [U5S:a([RSS RU535 ggs snf) zy Verify the TP plan of the model, log a warning if the layers that were not sharded and the rules that were not applied. NrQrrr;r<z>The following TP rules were not applied on any of the layers: z'The following layers were not sharded: z, ) rOsetcopyrRrLrMpopdiscardrrNrOjoin) expected_keysr9r' generic_keysunsharded_layers unused_rulesr@rUrDs rBverify_tp_planrnsA ERS]c4S9]LS<(<<>L.1SjSZZQ'*c VVFC<  (   / 6  $ $S ) & &ASAZAZ[^`aAbcdAe,e,=+q   . 5  $ $S ) <1WXdWefg q @K[A\@]^_!#TsE cX@lX0lUb;[U[5(a[R "U5nX R l[U[5(aXlURnUb[(aUR5H"nU[;dM[SUS[35e UR5H7upx[USS5(d[XuSS9n [!UUU UU5 SUlM9 U$)z,Distribute a model according to the TP plan.z"Unsupported tensor parallel style z. Supported styles are _is_hookedF)rSr9rTT)_tp_size _device_meshrddictr from_dictrdistributed_configr9r/valuesr3r named_modulesr'rWrUrp) rRr9rur;r: model_planvrNrplans rBdistribute_modelr{sN$% ($ / /!2!">""$A++ #EaSH_`s_t!uvv%"//1LD6<77-Tafg3 !%F 2 LrKr;)r9zstr | dict[str, str] | Noner:r>)rNr.r?r.)T)rSr.r9dict[str, str]r?z str | None)rgr*rhzint | list[int]r?z list[int])r) rr=rr*rr*rr*r?r=r)rr>r@)r5r=r6r*r;zdist.device_mesh.DeviceMeshr?r=)r>dict[str, torch.Tensor]r9r|r:r*r?r})rjz list[str]r9zdict[str, str] | None)Q __future__rrrr+rL functoolsrr(rutilsrr utils.genericr utils.import_utilsr r"torch.distributedrr is_availabler/ get_loggerrrNrCrOrWrduint8int8int16rvbfloat16int32float32float64r float8_e4m3fnrwrmrrrrautogradFunctionrrrrrrrrr rrrrErir~rrrrrrrrr rr3r1r9rFrUrcrnr{rrKrBrs3#  +6,3$$)#4#4#A#A#C    H %dh>, (>,3=>,BE. kkjj{{}}{{}}}}{{&& L&4A4P > "> > >  >  > Be>P.4!00!$ !// !/((/DGU^^ $ $GD&GU^^,,&G\4 3 , ( 0     "6 6 r/])/]d;"5;<-R,-R`?K)?KD8O8,8O84N_+N_b9*92*b-*bZT9(T9n$92$9NP 3P,?*(?*D*;)<&<66+.6=X66