Zj|SrSSKrSSKrSSKrSSKrSSKJr SSKJ r SSK J r J r J r \ "5(a SSKrSSKJr \ R "\5rSr\ "5(a\"5(aSS KJr OSS KJr "S S \5r"S S\5rSqSrSrSrSrSrSr SSjr!Sr"SSjr#SSjr$SSjr%Sr&g)z Integration with Deepspeed N) partialmethod)dep_version_check)is_accelerate_availableis_torch_availablelogging)nnc[RRS5SLnU(a![RRS5ngg![RRa gf=f)N deepspeedTF) importlibutil find_specmetadataPackageNotFoundError)package_exists_s t/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/integrations/deepspeed.pyis_deepspeed_availabler$sc^^--k:$FN ""++K8A!!66  sA A,+A,)HfDeepSpeedConfig)objectc,^\rSrSrSrU4SjrSrU=r$)r9a" This object contains a DeepSpeed configuration dictionary and can be quickly queried for things like zero stage. A `weakref` of this object is stored in the module's globals to be able to access the config from areas where things like the Trainer object is not available (e.g. `from_pretrained` and `_get_resized_embeddings`). Therefore it's important that this object remains alive while the program is still running. [`Trainer`] uses the `HfTrainerDeepSpeedConfig` subclass instead. That subclass has logic to sync the configuration with values of [`TrainingArguments`] by replacing special placeholder values: `"auto"`. Without this special logic the DeepSpeed configuration is not modified in any way. Args: config_file_or_dict (`Union[str, Dict]`): path to DeepSpeed config file or dict. cf>[U5 [S5 [S5 [TU] U5 g)N accelerater )set_hf_deepspeed_configrsuper__init__selfconfig_file_or_dict __class__s rrHfDeepSpeedConfig.__init__Js)%,'+& ,-)__name__ __module__ __qualname____firstlineno____doc__r__static_attributes__ __classcell__r!s@rrr9s ..r#rc`^\rSrSrSrU4SjrSrSrS Sjr\ "\SS9r S S jr S r S r U=r$)HfTrainerDeepSpeedConfigRz The `HfTrainerDeepSpeedConfig` object is meant to be created during `TrainingArguments` object creation and has the same lifespan as the latter. c@>[TU]U5 SUl/UlgN)rr_dtype mismatchesrs rr!HfTrainerDeepSpeedConfig.__init__Xs ,- r#cJURc [S5eUR$)Nz8trainer_config_process() wasn't called yet to tell dtype)r2 ValueError)rs rdtypeHfTrainerDeepSpeedConfig.dtype]s" ;; WX X{{r#c6URU5nUcgUS:H$)NFauto) get_value)r ds_key_longvals ris_auto HfTrainerDeepSpeedConfig.is_autobs"nn[) ;&= r#c URU5upVUcgURU5S:XaX%U'gU(dgURU5nUb.Xr:wa(URRSUSUSUSU35 ggg)a A utility method that massages the config file and can optionally verify that the values match. 1. Replace "auto" values with `TrainingArguments` value. 2. If it wasn't "auto" and `must_match` is true, then check that DS config matches Trainer config values and if mismatched add the entry to `self.mismatched` - will assert during `trainer_config_finalize` for one or more mismatches. Nr:z- ds =z vs hf )find_config_nodegetr3append)rr<hf_valhf_key must_matchconfigds_keyds_vals r fill_match#HfTrainerDeepSpeedConfig.fill_matchis..{; >  ::f  '#6N  F#  &"2 OO " "U;-qxqQWPX#Y Z#3 r#F)rGcURUR-UR-nURSURSU(+5 URSURS5 URSUSU(+5 URSURS5 URSUR S 5 URS UR UR/S 5 URS URS 5 URSURS5 URSS5 URSUR S 5 UR(aEURRS05URS'URURSS'URSUR=(d URS5 URSUR =(d UR"S5 UR%S5(a[&R(UlgUR%S5(a[&R,Ulg[&R.Ulg)zr Adjust the config with `TrainingArguments` values. This stage is run during `TrainingArguments` object creation. train_micro_batch_size_per_gpuper_device_train_batch_sizegradient_accumulation_stepstrain_batch_sizeztrain_batch_size (calculated)gradient_clipping max_grad_normzoptimizer.params.lr learning_ratezoptimizer.params.betaszadam_beta1+adam_beta2zoptimizer.params.eps adam_epsilonzoptimizer.params.weight_decay weight_decayzscheduler.params.warmup_min_lrrzscheduler.params.warmup_max_lr checkpointuse_node_local_storagez fp16.enabledzfp16|fp16_full_evalz bf16.enabledzbf16|bf16_full_evalN) world_sizerOrPrKrSrT adam_beta1 adam_beta2rUrV fill_onlysave_on_each_noderHrCfp16fp16_full_evalbf16bf16_full_evalis_truetorchbfloat16r2float16float32)rargsauto_find_batch_sizerQs rtrainer_config_process/HfTrainerDeepSpeedConfig.trainer_config_processs ??T-M-MMPTPpPpp  ,  , , )$ $   )  , , )    +$ $  +T-?-?Q -t/A/A?S  $ __doo . # .0A0A>R 79J9JN[ 7; 8$:L:Lo^  ! !(,  b(IDKK %BFBXBXDKK %&> ? )Id6I6ILab )Id6I6ILab << ' '..DK \\. ) )--DK--DKr#c/SQnUVs/sHoPRU5(dMUPM nn[U5S:GaSn[US5(Ga8[URS5(aURRnGO[URS5(a [ URR 5nO[URS5(aF[URRS5(a!URRRnOi[URS5(aN[URRS5(a)[ URRR 5nUc[SUS 35eURS Xw-5 UR5(a6URS [S U-U-55 URS SU-5 URSUS5 URSURU5S5 [UR5S:a*SRUR5n[SUS35egs snf)zx This stage is run after we have the model and know num_training_steps. Now we can complete the configuration process. )$zero_optimization.reduce_bucket_size-zero_optimization.stage3_prefetch_bucket_size4zero_optimization.stage3_param_persistence_thresholdrNrH hidden_size hidden_sizes text_configzThe model's config file has neither `hidden_size` nor `hidden_sizes` entry, therefore it's not possible to automatically fill out the following `auto` entries in the DeepSpeed config file: zb. You can fix that by replacing `auto` values for these keys with an integer value of your choice.rlrmg?rn z scheduler.params.total_num_stepsznum_training_steps (calculated)z!scheduler.params.warmup_num_steps warmup_steps z]Please correct the following DeepSpeed config values that mismatch TrainingArguments values: zF The easiest method is to set these DeepSpeed config values to 'auto'.)r>lenhasattrrHromaxrprqr6r\is_zero3intrKget_warmup_stepsr3join) rrgmodelnum_training_stepshidden_size_based_keysxhidden_size_auto_keysror3s rtrainer_config_finalize0HfTrainerDeepSpeedConfig.trainer_config_finalizes "  -C V,BqllSTo,B V $ % )Kuh''5<<77"',,":":KU\\>::"%ell&?&?"@KU\\=99gellF^F^`m>n>n"',,":":"F"FKU\\=99gellF^F^`n>o>o"%ell&>&>&K&K"LK" 55J4KLYY NNA;C\ ]}}Ck)K78J$  .  -  /  ! !"4 5  t ! #4??3J'L(oq  $a!Ws I4I4)r2r3)NTF)r%r&r'r(r)rr7r>rKrr\rirr*r+r,s@rr.r.Rs=   ![4jU;I8(tCCr#r.c0[R"U5qgr1)weakrefref_hf_deepspeed_config_weak_ref)hf_deepspeed_config_objs rrr s %,KK0G$H!r#cSqgr1)rr$r#runset_hf_deepspeed_configrs %)!r#cX[b#[5b[5R5$g)NF)rrxr$r#ris_deepspeed_zero3_enabledrs&$05R5T5`,.7799r#cP[b[5b[5R$gr1)rrHr$r#rdeepspeed_configrs#$05R5T5`,.555r#c"^^^^SSKmSSKnSSKJn SSKJm UR 5mUUUU4SjmUR"5 U"5 T"XR5 SSS5 SSS5 g!,(df  N=f!,(df  g=f)a- DeepSpeed ZeRO-3 variant of `PreTrainedModel.initialize_weights`. Mirrors the `smart_apply` dispatch logic but gathers each module's partitioned parameters before calling `_initialize_weights`, so initialization operates on full tensors instead of empty shards. Only rank 0 performs the actual init. rNr)guard_torch_init_functions)PreTrainedModelc>UR5H0n[UT5(aT"X"R5 M(T"X!5 M2 [UR SS95nU(aKTR R USS9 TRR5S:Xa U"UT5 SSS5 gU"UT5 g!,(df  g=f)NF)recurser modifier_rank) children isinstance_initialize_weightslist parameterszeroGatheredParameterscommget_rank)model_or_modulefnchildparamsr _apply_zero3r is_remote_codes rr.initialize_weights_zero3.._apply_zero34s$--/E%11U$=$=>U' 0 o000?@ 2262K>>**,17LK  / LKs ?(B:: C) r rcinitializationrmodeling_utilsrrno_gradr)r|rcrrrr rs @@@@rinitialize_weights_zero3r%sc;0))+N 0 0  ' )  9 9 :*  ) ) s$B A/B/ A= 9B BcZ^![5nUbURS05RSS5nURS05n[U[5(a+[ XERS05RSS55nUS:a [ S5eSS KJnJnJ m!J n [US S5n URn 0n UR5R5H1up[R "U R"U R$S S 9X'M3 UVs/sHn[X5(dMUPM nnUVs/sHn[X5(dMUPM nn['U5S :XaC0nUR5H!unnU"UU/X5unnUU ;dMUUU'M# U bU UlU$UVVs0sHnUR*HnUU_M M nnn0n0n[-UR/5U!4SjS9nUHnUR1U5nU"UUUX5unnUU ;dM*UbSUUnU"UR*UR2UR4S9nUR7UU5nUR9UUUU5 MUUU'M UR5H]unnUR;UUUR<S9nUR5H'unn [U [>5(aU S OU n U UU'M) M_ U bU UlU$s snfs snfs snnf![@an [CSUSU 35U eSn A ff=f)z Apply weight conversions (renaming and merging/splitting operations) to a state dict. This is a simplified version that handles the conversion without loading into the model. Ntensor_parallel autotp_size inferencetp_sizezWeight conversions (e.g., MoE expert fusion) with DeepSpeed Tensor Parallelism are not yet implemented but support is coming soon. Please disable tensor_parallel in your DeepSpeed config or convert your checkpoint to the expected format first.r)WeightConverterWeightRenamingdot_natural_keyrename_source_key _metadatameta)r7devicerc>T"U5$r1r$)krs r9_apply_weight_conversions_to_state_dict..s /!:Lr#)key)source_patternstarget_patterns operations)r|rHz'Failed to apply weight conversion for 'zb'. This likely means the checkpoint format is incompatible with the current model version. Error: )"rrCrdictrwNotImplementedErrorcore_model_loadingrrrrgetattrbase_model_prefix state_dictitemsrcemptyshaper7rurrsortedkeyspoprr setdefault add_tensorconvertrHr Exception RuntimeError)"r|rweight_mapping ds_configrinference_configrrrrprefixmodel_state_dictrparamentry renamings convertersnew_state_dict original_keytensor renamed_keyr converterrpattern_to_converterconversion_mapping sorted_keyssource_pattern new_convertermappingrealized_value target_nameers" @r'_apply_weight_conversions_to_state_dictrHsg!"I-- 126::=!L$==b9 & - -'#7#78I2#N#R#RS\^_#`aG Q;%d  ihz;5H  $ $F&&(..0  % EKKu{{SY Z1%3XN5j6WNIX%3Z^Ez%7Y%^JZ :!$.$4$4$6 L&.|YFeNK...4{+%7  '/N $;Eh*YiNgNgAyLNgA*h N*0LMK#  -&7 iQ[]c&v# ^ * *)1@ /$-$=$=$-$=$=(33! -77 ]S""; nfU/5{++$0!3 8 8 : W $__||-N '5&:&:&<" U$.ud$;$;a.3{+'=!;$#+ KYZiT 9+G   s7K9*K96K>K>-!LAL  L*L%%L*c ^^ ^ ^ [USS5m UR5nT bT UlSnUb [USS5nUb![U5S:a[ XU5nX0l/mUR 5n[UR55m [USS5nUR5VVs0sH&upgURUSU35bUSU3OUU_M( nnnS S[R4UU U U 4S jjjm T "XSS 9 TT 4$s snnf) a Loads state dict into a model specifically for Zero3, since DeepSpeed does not support the `transformers` tensor parallelism API. Nearly identical code to PyTorch's `_load_from_state_dict` Args: model_to_load: The model to load weights into state_dict: The state dict containing the weights load_config: Optional LoadStateDictConfig containing weight_mapping and other loading options rNrrr.Fmodulec>Tc0OTRUSS05nX4S'XUS//T4n[5(GaLSSKn[UR USSSS95n/nUH7n X;dM Xyn SU lUR U 5 TRU 5 M9 [U5S:aTURRUSS9 [RR5S:XaUR"U6 SSS5 [URUSSSS95n U R!5HZupX;dM U cMTRU 5 [R""5 U R%X5 SSS5 SU lM\ UR&R!5HupUcM T"XX--S-U5 M g!,(df  N=f!,(df  Nb=f) Nassign_to_params_buffersTrF)rrrr)rCrr rnamed_parameters_is_hf_initializedrDdiscardrurrrc distributedr_load_from_state_dict named_buffersrrcopy__modules)rrrrlocal_metadatargr rparams_to_gatherrrrbufnamer error_msgsloadr missing_keyss rr/_load_state_dict_into_zero3_model..loads'/X\\&"+r5R5M12ND"b*M & ' '  $F$;$;6#2;X]$;$^_ ! %?,/E/3E,$++E2 ((+ &#$q(^^667GWX6Y((113q844d;Z !!5!5VCR[RW!5!XYM'--/?s ((+ *-0)-1C* 0"??002KD U (;=UV3ZY)s2F<G < G  G )r)F) rcopyrrur_weight_conversionsrsetrrrCr Module) model_to_loadr load_configrmeta_model_state_dict prefix_modelrvrrrrs @@@@r!_load_state_dict_into_zero3_modelrs9z;5H"J' N .>E!c.&9A&=<]Xfg ,:)J)446,1134L=*=tDL$$&&DA#8";";|nAaS1# dehi i&)WRYY)W)WV UC | ##is--Dc0^^SSKJnJn URnSnSU;aU"US9nO?UR 5(a[ R S5 TR5nSUS'Sn S U;a U"U5n X4$[X5(a UU4S jn U"XS 9n X4$) zQ A convenience wrapper that deals with optimizer and lr scheduler configuration. r) DummyOptimDummySchedulerN optimizer)rzDetected ZeRO Offload and non-DeepSpeed optimizers: This combination should work as long as the custom optimizer has both CPU and GPU implementation (except LAMB)Tzero_allow_untested_optimizer schedulercb>[R"T5nSUlURTUS9nU$)N)r}r )r lr_schedulercreate_scheduler)r  trainer_copyrr}trainers r_lr_scheduler_callable5deepspeed_optim_sched.._lr_scheduler_callable3s=#yy1 -1 )+<<'9Y =  $#r#)lr_scheduler_callable) accelerate.utilsr r rH is_offloadloggerinfocreate_optimizerr) rhf_deepspeed_configrgr}model_parametersr r rHr rrs ` ` rdeepspeed_optim_schedr s< ' 'FIf&67  ) ) + + KKV ,,. 26./Lf%i0 "  "" i , , $*)bL  ""r#cSSKJn URnURnURR R RnURXTU5 URUR55 U(aKUR5(d [S5eURS5 URS5 SupxSn Xx4$SUlURR!S05R!S S 5n U S :a.SSKn U R%UU UR'5URS 9n[)[+S UR-555n [/XXQU 5upxXx4$) a Init DeepSpeed, after updating the DeepSpeed configuration with any relevant Trainer's args. If `resume_from_checkpoint` was passed then an attempt to resume from a previously saved checkpoint will be made. Args: trainer: Trainer object num_training_steps: per single gpu resume_from_checkpoint: path to a checkpoint if to resume from after normal DeepSpeedEngine load inference: launch in inference mode (no optimizer and no lr scheduler) auto_find_batch_size: whether to ignore the `train_micro_batch_size_per_gpu` argument as it's being set automatically by the auto batch size finder Returns: optimizer, lr_scheduler We may use `deepspeed_init` more than once during the life of Trainer, when we do - it's a temp hack based on: https://github.com/deepspeedai/DeepSpeed/issues/1394#issuecomment-937405374 until Deepspeed fixes a bug where it can't resume from a checkpoint after it did some stepping https://github.com/deepspeedai/DeepSpeed/issues/1612 r)rzMZeRO inference only makes sense with ZeRO Stage 3 - please adjust your configr r)NNNrrr)r|rr7rHcUR$r1) requires_grad)ps rr deepspeed_init..{sr#)deepspeed.utilsrr|rg acceleratorstatedeepspeed_plugin hf_ds_configrsetLevelget_process_log_levelrxr6del_config_sub_treer rHrCr tp_model_initr7rfilterrr) rr}r ds_loggerr|rgrr rrdeepspeed_tp_sizer s rdeepspeed_initr0Csd*4 MME <[R SU35 UR UUSSS9upVUc[ SU35eg[ SU35e)Nrz /global_step*zAttempting to resume from T)load_module_strictload_optimizer_statesload_lr_scheduler_statesz-[deepspeed] failed to resume from checkpoint z!Can't find a valid checkpoint at )globrrurrload_checkpointr6)deepspeed_enginecheckpoint_pathr2r5deepspeed_checkpoint_dirs load_pathrs rdeepspeed_load_checkpointr;s  &tyyO3DM1R'S T $%) 00ABC'77 1"&%) 8  L_L]^_ _ <_sV!!22I5i6L6L6S6STI!*!7!7!>!>I 11$Mr#c^ ^SU;aSU;aUSUS'U"S 0UD6nURnURS:Xa'URS:aSSKJn UR 5nO6UR bUR SR5nO [S5eURn [RRRRXhS 9mUSS :gRS 5R5n [RRRRXS 9m [U U4S j[!U 555n [T 5n U [#U S5- nU(aXe4$U$)aA Computes the loss under sequence parallelism with `sp_backend="deepspeed"` and `sp_size > 1`. Performs weighted loss aggregation across SP ranks, accounting for varying numbers of valid tokens per rank (e.g., when some ranks receive only padding or prompt tokens that are masked with -100). Args: accelerator (`Accelerator`): The accelerator instance with `torch_device_mesh` support. model (`torch.nn.Module`): The model to compute the loss for. inputs (`dict[str, torch.Tensor | Any]`): The input data for the model. Must include `"shift_labels"` key. return_outputs (`bool`): Whether to return the model outputs along with the loss. pc (`accelerate.parallelism_config.ParallelismConfig`): The parallelism configuration. Returns: The loss, or a tuple of `(loss, outputs)` if `return_outputs` is `True`. labels shift_labelsr rr)groupsspzSequence parallelism is enabled but no SP process group is available. Ensure torch_device_mesh is initialized or sp_backend='deepspeed' with sp_size > 1.)groupirc3P># UHnTUS:dMTUTU-v M g7f)rNr$).0rankgood_tokens_per_ranklosses_per_ranks r ,deepspeed_sp_compute_loss..s7(D  % ) ; 4T ::(s &&r$)loss sp_backendsp_sizer$rB_get_sequence_parallel_grouptorch_device_mesh get_groupr6rcrr functional all_gatherviewsumrangerw)r%r|inputsreturn_outputspcoutputsrLrBsp_group sp_world_size good_tokens total_losstotal_good_tokensrHrIs @@rdeepspeed_sp_compute_lossr`s\,v.F":!.1xofoG <r`r$r#rrjs #9HH   H % !7!9!9O3..2p0ph!%I)  ;FhVW$t3#l@#FP0N"77r#