ZjR^SSKJr SSKJr SSKrSSKJr SSKJr SSKJ r J r SSK J r SS K JrJr SS KJrJr SS KJrJrJrJr SS KJrJr SS KJrJr SSKJrJ r SSK!J"r" SSK#J$r$J%r%J&r& SSK'J(r(J)r) SSK*J+r+ SSK,J-r- \"S5"SS\R\55r/"SS\R\5r0Sr1\"S5S=Sj5r2S\RfS\4S\Rf4S jr5S>S!\R\S"\RfS#\RfS$\RfS%\RfS-S&\6S'\6S(\"\$4S)jjr7"S*S+\R\5r8"S,S-\R\5r9"S.S/\5r:\%"S0S1\ 55r;\%"S2S3\;55r<\%"S4S5\;\ 55r="S6S7\\;5r>"S8S9\\;5r?"S:S;\\;5r@/S[TU]5 [R"[R "U55UlX lg)z- Exaone4RMSNorm is equivalent to T5LayerNorm N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizer& __class__s }/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/exaone4/modeling_exaone4.pyr*Exaone4RMSNorm.__init__3s/ ll5::k#:; # hidden_statescURnUR[R5nUR S5R SSS9nU[R "X0R-5-nURURU5-$)NT)keepdim) dtypetor,float32powmeanrsqrtr/r.)r0r6 input_dtypevariances r3forwardExaone4RMSNorm.forward;sw#)) %((7  $$Q',,R,>% H?T?T4T(UU {{]--k:::r5c^[URR5SUR3$)Nz, eps=)tupler.shaper/)r0s r3 extra_reprExaone4RMSNorm.extra_reprBs* ))*+6$2G2G1HIIr5)r/r.)gư>) __name__ __module__ __qualname____firstlineno__floatr*r,TensorrCrH__static_attributes__ __classcell__r2s@r3r$r$1sB$$$$$;U\\;ell;JJr5r$c ^\rSrSr%\R \S'SS\4U4Sjjjr\ SS\S-S\ SS\ S-S \ S \ 44S jj5r\R"5\S 55rS rU=r$)Exaone4RotaryEmbeddingFinv_freqNconfigc>[TU]5 URUlURUlXlUR R SUlURnURS:wa[URnU"UR U5uo@l URSUSS9 URSUR5SS9 g)N rope_typedefaultrVF) persistentoriginal_inv_freq) r)r*max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrWrope_parametersrYcompute_default_rope_parametersrattention_scalingregister_bufferclone)r0rWdevice rope_init_fnrVr2s r3r*Exaone4RotaryEmbedding.__init__Is "("@"@$*$B$B! 44[A!%!E!E >>Y &.t~~>L+7 V+L(( ZeD 0(..2BuUr5rez torch.deviceseq_lenr'z torch.Tensorc URSn[USS5=(d URUR-nSnSU[R "SUS[R S9RU[RS9U- -- nXe4$) aH Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetahead_dimNg?rr8r;)rer;) r`getattrr1num_attention_headsr,arangeint64r<rN)rWrerhbasedimattention_factorrVs r3ra6Exaone4RotaryEmbedding.compute_default_rope_parametersYs&%%l3fj$/c63E3EIcIc3c U\\!S!5;;?BB&X]XcXcBdgjj k ))r5cLURSSS2S4R5RURSSS5R UR 5nUSS2SSS24R5n[ UR R[5(a0UR RS:waUR ROSn[USS9 UR5UR5-RSS5n[R"Xf4SS 9nUR5UR-nUR5UR-n SSS5 WR UR S 9W R UR S 94$!,(df  N@=f) Nrr9r mpscpuF) device_typeenabledr8rrrl)rVrNexpandrGr<re isinstancetypestrr transposer,catcosrbsinr;) r0x position_idsinv_freq_expandedposition_ids_expandedrxfreqsembrrs r3rCExaone4RotaryEmbedding.forwardwsN!MM$4-8>>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E'E!((--[`J`ahhmmfk  U C&,,.1F1L1L1NNYYZ[]^_E))UN3C'')d444C'')d444C D vvAGGv$cff177f&;;; D Cs BF F#)rbrWr^r_rYN)NNN)rJrKrLrMr,rO__annotations__r!r* staticmethodrintrFrNrano_gradrrCrPrQrRs@r3rTrTFsllV}VV '++/"*$*(*t* ~u$ % **: ]]_ <