Zj/$ SSKrSSKrSSKJr SSKJr SSKr\RRR\RRR\RRR\RRR\RRR\RRR\RRR\RRR \RRR"\RRR$\RRR&\RRR(\RRR*\RRR,S.rS*S\R0S\S\S\R4S-S \R04 S jjr S*S\R0S \S \S\R4S-S \R04 S jjr S\R0S\S \R04Sjr S\R0S \R04Sjr S\R0S \R04Sjr S\R0S \R04SjrS+S\R0S\S \R04SjjrS,S\R0S\S\R4S-S \R04SjjrS,S\R0S\S\R4S-S \R04SjjrS-S\R0S\S\S\S\R4S-S \R04 SjjrS-S\R0S\S\S\S\R4S-S \R04 SjjrS.S\R0S \S \S\S\S\R4S-S \R04SjjrS/S\R0S\S\R4S-S \R04SjjrS0S\R0S\S \S\R4S-S \R04 SjjrS\R0S \R0S \R04S!jrS1S"jrS#rS$r S%r!\S&5r"\S'5r#\S(5r$\S)5r%g)2N) defaultdict)contextmanager)uniform_normal_ constant_ones_zeros_eye_dirac_xavier_uniform_xavier_normal_kaiming_uniform_kaiming_normal_ trunc_normal_ orthogonal_sparse_tensorab generatorreturncF[USS5(d[S"XX#S9$U$)N_is_hf_initializedFr)rrrgetattrTORCH_INIT_FUNCTIONS)rrrrs l/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/initialization.pyrr*s+ 6/ 7 7#J/qVV MmeanstdcF[USS5(d[S"XX#S9$U$)NrFr)rr rr)rrr rs rrr2s+ 6/ 7 7#I.vc__ MrvalcD[USS5(d [S"XS9$U$)NrFr)r"r)rr"s rrr:s' 6/ 7 7#K0AA MrcH[USS5(d[S"U5$U$)NrFrrrs rrr@s' 6/ 7 7#G,V44 MrcH[USS5(d[S"U5$U$)NrFr rr%s rr r Fs' 6/ 7 7#H-f55 MrcH[USS5(d[S"U5$U$)NrFr rr%s rr r Ls' 6/ 7 7#F+F33 MrgroupscD[USS5(d [S"XS9$U$)NrFr )r(r)rr(s rr r Rs' 6/ 7 7#H-fDD MrgaincF[USS5(d[S"XUS9$U$)NrFr r*rrrr*rs rr r Xs+ 6/ 7 7#$56vT]^^ MrcF[USS5(d[S"XUS9$U$)NrFr r,rr-s rr r ^s+ 6/ 7 7#$45fS\]] Mrmode nonlinearitycH[USS5(d[S"XX#US9$U$)NrFrrr/r0rrrrr/r0rs rrrds5 6/ 7 7#$67 d   MrcH[USS5(d[S"XX#US9$U$)NrFrr2rr3s rrrrs5 6/ 7 7#$56 d   Mrc H[USS5(d[S"XX#XES9$U$)NrFr)rr rrrr)rrr rrrs rrrs. 6/ 7 7#O4VCXYoo MrcF[USS5(d[S"XUS9$U$)NrFrr,rr-s rrrs, 6/ 7 7#M26PYZZ MrsparsitycF[USS5(d[S"XX#S9$U$)NrFr)r7r rr)rr7r rs rrrs+ 6/ 7 7#I.vcgg Mrotherc[USS5(d0[R"5 URU5sSSS5 $U$!,(df  U$=f)NrF)rtorchno_gradcopy_)rr9s rr=r=s@ 6/ 7 7 ]]_<<&_ M_ Ms A Ac[RRRU5up4US:XaUnOUS:XaUnO US:XaX4-S- nSW- nUS:Xa"[ U[ R "U5S- S9 gUS :Xa[U[ R "U5S9 gUS :Xa'[ R "S U-5n[X*U5 g[S U35e) Nfan_infan_outfan_avg?truncated_normalg۶%?)r normaluniformzinvalid distribution ) r;nninit_calculate_fan_in_and_fan_outrmathsqrtrr ValueError)rr/ distributionr?r@denomvariancebounds r_variance_scalingrRshhmmAA&IOF x    !Q&U{H))f$))H"58K"KL  !DIIh/0  " !h,''0?@@rc@[USS5(d [USSS9 U$)NrFr?rDr/rNrrRr%s r lecun_normal_rVs$ 6/ 7 7&x>PQ Mrc@[USS5(d [USSS9 U$)NrFr?rErTrUr%s rdefault_flax_embed_init_rXs# 6/ 7 7&xhG Mr) z torch.nn.initztorch.nn.modules.activationztorch.nn.modules.transformerztorch.nn.modules.linearztorch.nn.modules.lossztorch.nn.modules.batchnormztorch.nn.modules.convztorch.nn.modules.normalizationztorch.nn.modules.rnnztorch.nn.modules.sparsec #J# [[5n[HnU[R;dM[RUn[ R 5H<n[X#5(dM[X#5XU'[X#[5U5 M> M Sv UR5H*up$UR5Hup5[X#U5 M M, g!UR5H*up$UR5Hup5[X#U5 M M, f=f7f)av Guard the `torch.nn.init` primitive functions to behave exactly like the functions in this file, i.e. be protected against the `_is_hf_initialized` flag to avoid re-init if the param was already loaded. Usually, all models are using the init from `transformers` which are already guarded, but just to make extra sure and for remote code, we also use this context manager. N) rdictTORCH_MODULES_TO_PATCHsysmodulesrkeyshasattrrsetattrglobalsitems) originals module_namemodule func_name functionsfuncs rguard_torch_init_functionsrisD!I11Kckk)[1!5!:!:!v7Q ))479Y3GH"=2 "+!2 F#,??#4 40$5"3!2 F#,??#4 40$5"3s(D#C 8C .3C !?D# AD  D#c ## SSKJn Sn[[5n[HunU[ R ;dM[ R Un[R5H1n[XE5(dM[XE5X$U'[XEU5 M3 Mw URnXl Sv UR5H*upGUR5HupX[XEU5 M M, X`l g!UR5H*upGUR5HupX[XEU5 M M, WUl f=f7f)aS Disable weight initialization both at the torch-level, and at the transformers-level (`init_weights`). This is used to speed-up initializing an empty model with deepspeed, as we do not initialize the model on meta device with deepspeed, but we still don't need to run expensive weight initializations as we are loading params afterwards. PreTrainedModelcgNargskwargss r empty_func#no_init_weights..empty_func rN)modeling_utilsrmrrZr[r\r]rr^r_rr` init_weightsrb) rmrtrcrdrerforiginal_init_weightsrgrhs rno_init_weightsrzs0 D!I=1Kckk)[1!5!:!:!v7Q ))4:>"=2!0 < <'1$ "+!2 F#,??#4 40$5"3(=$ "+!2 F#,??#4 40$5"3(=$s)EC68C67:C61AE6AD==Ec#p# SSKJn SnURnXlSv X lg!WUlf=f7f)a Disable weight tying during loading with `from_pretrained`. This is needed as we want to have access to ALL weights in the state_dict during `from_pretrained`, and otherwise tying them would remove them from it, as it's called in `post_init` when instantiating. rkrlcgrorprqs rrt"no_tie_weights..empty_func$rvrN)rwrm tie_weights)rmrtoriginal_tie_weightss rno_tie_weightsrs:0 ;.::&0# ';#&:#s 6*6 36c#^# [RmU4SjnU[lSv T[lg!T[lf=f7f)av During meta-device model initialisation, ``torch.linspace`` produces meta tensors that have no data. Custom models loaded from the Hub (remote code) often call ``.item()`` on these tensors to compute scalar hyperparameters (e.g. stochastic-depth / drop-path schedules). Native transformers models already pass ``device="cpu"`` explicitly for such calls (see e.g. ``modeling_swin.py``, ``modeling_pvt_v2.py``), but remote-code models written before v5 do not. This context manager patches ``torch.linspace`` to default to ``device="cpu"`` when no explicit device is requested, matching the best practice already used throughout transformers. Calls that supply an explicit ``device`` argument (e.g. ``device=self.logits.device``) are left untouched. ``torch.arange`` is intentionally NOT patched because it is used in RoPE computations where the device must match model parameters. c8>URSS5 T"U0UD6$)Ndevicecpu) setdefault)rrrsoriginal_linspaces r_safe_linspace5meta_device_safe_creation_ops.._safe_linspaceEs#(E* $1&11rN)r;linspace)rrs @rmeta_device_safe_creation_opsr1s7$2$EN+ **s"A6 A AA)rCN)rk)rCN)rr? leaky_reluN)rrCgg@N)rkN)g{Gz?N)r?rE)&rKr\ collectionsr contextlibrr;rHrIrrrrr r r r r rrrrrrTensorfloat Generatorintstrr=rRrVrXr[rirzrrrprrrs #%  &&xx}}$$(( XX]] hhmm"" HHMM  hhmm""xx}}44hhmm22 66xx}}44XX]]0088==,,xx}}$$&_c LL"-2EJ__W[E[ \\dh LL %27JO//\`J` \\ell5<< %,,5<< 5<<ELL %,, 5<<U\\ ELLZ^H^jojvjv 5<<uuY]G]iniuiu$(,  LL       %   \\  $(,  LL       %   \\  (,  LL         %  \\ (, LL % \\ cg LL$)05IN[_I_ \\%,,u|| A,  114!=!=H;;*++r