Zj4 HSSKrSSKrSSKJr SSKrSSKJrJr SSKJr SSK J r SSK J r \ R"\5r\"S5"S S \R 55r\r\"S 5"S S \R 55r\"S5"SS\R 55r\"S5"SS\R 55r\"S5"SS\R 55r\"S5"SS\R 55r"SS\R 5r"SS\R 5r"SS\R 5r"S S!\R 5r"S"S#\R 5r"S$S%\R 5r"S&S'\R 5r"S(S)\5r"S*S+\R 5r 0S,\_S-\S.S/S0.4_S1\_S2\_S3\S4S504_S6\_S7\S8S504_S9\_S:\RB_S;\_S<\RD_S=\_S>\_S?\_S@\RF_SA\_SB\RH_\RJ\\\RL\RN\RP\ SC.Er)\"\)5r*SDr+\+"S35r,\+"S25r-\+"S,5r.\+"S15r/\+"S65r0\+"S?5r1\+"SE5r2\+"S>5r3\+"S=5r4g)FN) OrderedDict)Tensornn)use_kernel_forward_from_hub)logging)is_torchdynamo_compilingGeluTanhc\^\rSrSrSrS S\4U4SjjjrS\S\4SjrS\S\4Sjr S r U=r $) GELUTanha A fast C implementation of the tanh approximation of the GeLU activation function. See https://huggingface.co/papers/1606.08415. This implementation is equivalent to NewGELU and FastGELU but much faster. However, it is not an exact numerical match due to rounding errors. use_gelu_tanh_pythonc>[TU]5 U(aURUlg[R "[ RRSS9Ulg)Ntanh) approximate) super__init___gelu_tanh_pythonact functoolspartialr functionalgelu)selfr __class__s i/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/activations.pyrGELUTanh.__init__(s<  --DH ((););PDHinputreturnc US-S[R"[R"S[R- 5US[R "US5---5--$N??@Hm?g@torchrmathsqrtpipowrrs rrGELUTanh._gelu_tanh_python/sPs{cEJJtyytww/G5S[^c^g^ghmor^sSsKs/t$uuvvrc$URU5$Nrr-s rforwardGELUTanh.forward2xxrr1F) __name__ __module__ __qualname____firstlineno____doc__boolrrrr2__static_attributes__ __classcell__rs@rr r sJQTQQwvw&wVrr NewGELUc*\rSrSrSrS\S\4SjrSrg)NewGELUActivation:z Implementation of the GELU activation function currently in Google BERT repo (identical to OpenAI GPT). Also see the Gaussian Error Linear Units paper: https://huggingface.co/papers/1606.08415 rr c SU-S[R"[R"S[R- 5US[R "US5---5--$r"r'r-s rr2NewGELUActivation.forwardAsPU{cEJJtyytww/G5S[^c^g^ghmor^sSsKs/t$uuvvrNr6r7r8r9r:rr2r<rErrrArA:s wVwwrrAGeLUc\^\rSrSrSrS S\4U4SjjjrS\S\4SjrS\S\4Sjr S r U=r $) GELUActivationEa Original Implementation of the GELU activation function in Google BERT repo when initially created. For information: OpenAI GPT's GELU is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) This is now written in C in nn.functional Also see the Gaussian Error Linear Units paper: https://huggingface.co/papers/1606.08415 use_gelu_pythonc>[TU]5 U(aURUlg[R R Ulgr0)rr _gelu_pythonrrrr)rrKrs rrGELUActivation.__init__Ns/  ((DH}}))DHrrr cnUS-S[R"U[R"S5- 5--$)Nr#r$r%)r(erfr)r*r-s rrMGELUActivation._gelu_pythonUs,s{cEIIediin.D$EEFFrc$URU5$r0r1r-s rr2GELUActivation.forwardXr4rr1r5) r6r7r8r9r:r;rrrMr2r<r=r>s@rrIrIEsG***G&GVGVrrISiLUc*\rSrSrSrS\S\4SjrSrg)SiLUActivation\a See Gaussian Error Linear Units (Hendrycks et al., https://arxiv.org/abs/1606.08415) where the SiLU (Sigmoid Linear Unit) was originally introduced and coined, and see Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning (Elfwing et al., https://arxiv.org/abs/1702.03118) and Swish: a Self-Gated Activation Function (Ramachandran et al., https://arxiv.org/abs/1710.05941v1) where the SiLU was experimented with later. rr c@[RRU5$r0)rrsilur-s rr2SiLUActivation.forwardfs}}!!%((rrENrFrErrrVrV\s)V))rrVFastGELUc*\rSrSrSrS\S\4SjrSrg)FastGELUActivationjzu Applies GELU approximation that is slower than QuickGELU but more accurate. See: https://github.com/hendrycks/GELUs rr c^SU-S[R"US-SSU-U---5--$)Nr#r$g3E?r&)r(rr-s rr2FastGELUActivation.forwardps:U{cEJJu|/CsXX]M]`eMeGe/f$gghhrrENrFrErrr]r]jsiViirr] QuickGELUc*\rSrSrSrS\S\4SjrSrg)QuickGELUActivationtzj Applies GELU approximation that is fast but somewhat inaccurate. See: https://github.com/hendrycks/GELUs rr c:U[R"SU-5-$)NgZd;?)r(sigmoidr-s rr2QuickGELUActivation.forwardzsu}}UU]333rrENrFrErrrcrcts4V44rrccJ^\rSrSrSrS\S\4U4SjjrS\S\4SjrS r U=r $) ClippedGELUActivation~ar Clip the range of possible GeLU outputs between [min, max]. This is especially useful for quantization purpose, as it allows mapping negatives values in the GeLU spectrum. For more information on this trick, please refer to https://huggingface.co/papers/2004.09602. Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))). See https://huggingface.co/papers/1606.08415 minmaxch>X:a[SUSUS35e[TU] 5 XlX lg)Nzmin should be < max (got min: z, max: )) ValueErrorrrrkrl)rrkrlrs rrClippedGELUActivation.__init__s8 9=cU'#aPQ Q rxr cl[R"[U5URUR5$r0)r(cliprrkrl)rrqs rr2ClippedGELUActivation.forwards!zz$q'488TXX66r)rlrk) r6r7r8r9r:floatrrr2r<r=r>s@rriri~s3 E77F77rric>^\rSrSrSrU4SjrS\S\4SjrSrU=r $)AccurateGELUActivationz Applies GELU approximation that is faster than default and more accurate than QuickGELU. See: https://github.com/hendrycks/GELUs Implemented along with MEGA (Moving Average Equipped Gated Attention) cz>[TU]5 [R"S[R- 5Ulg)N)rrr)r*r+precomputed_constantrrs rrAccurateGELUActivation.__init__s' $(IIa$''k$:!rrr c SU-S[R"URUS[R"US5---5--$)Nr#rr&)r(rr{r,r-s rr2AccurateGELUActivation.forwardsEU{a%**T-F-F%RZ]b]f]fglno]pRpJp-q"rrssr)r{) r6r7r8r9r:rrr2r<r=r>s@rrwrws);tVtttrrwcP^\rSrSrSrU4SjrS\S\4SjrS\S\4SjrSr U=r $) MishActivationz See Mish: A Self-Regularized Non-Monotonic Activation Function (Misra., https://huggingface.co/papers/1908.08681). Also visit the official repository for the paper: https://github.com/digantamisra98/Mish c`>[TU]5 [RRUlgr0)rrrrmishrr|s rrMishActivation.__init__s ==%%rrr cnU[R"[RR U55-$r0)r(rrrsoftplusr-s r _mish_pythonMishActivation._mish_pythons%uzz"--"8"8"?@@@rc$URU5$r0r1r-s rr2MishActivation.forwardr4rr1) r6r7r8r9r:rrrr2r<r=r>s@rrrs; &A&AVAVrrc*\rSrSrSrS\S\4SjrSrg)LinearActivationzS Applies the linear activation function, i.e. forwarding input directly to output. rr cU$r0rEr-s rr2LinearActivation.forwards rrENrFrErrrrsVrrc"\rSrSrSrSSjrSrg)LaplaceActivationz Applies elementwise activation based on Laplace function, introduced in MEGA as an attention activation. See https://huggingface.co/papers/2209.10655 Inspired by squared relu, but with bounded range and gradient for better stability cX- RU[R"S5-5nSS[R"U5--$)Nr%r#r$)divr)r*r(rP)rrmusigmas rr2LaplaceActivation.forwards:  3!78cEIIe,,--rrEN)g۞?g ^/ ?r6r7r8r9r:r2r<rErrrrs .rrc\rSrSrSrSrSrg)ReLUSquaredActivationzV Applies the relu^2 activation introduced in https://huggingface.co/papers/2109.08668 cp[RRU5n[R"U5nU$r0)rrrelur(square)rr relu_appliedsquareds rr2ReLUSquaredActivation.forwards)}}))%0 ,,|,rrENrrErrrrs rrc\rSrSrSrSrSrg)SqrtSoftplusActivationuFsqrt(softplus(x)) — the router scoring function used by DeepSeek V4.c\[RRU5R5$r0)rrrr*r-s rr2SqrtSoftplusActivation.forwards }}%%e,1133rrENrrErrrrs P4rrc(^\rSrSrU4SjrSrU=r$)ClassInstantiercl>[TU]U5n[U[5(aUOU04up4U"S0UD6$)NrE)r __getitem__ isinstancetuple)rkeycontentclskwargsrs rrClassInstantier.__getitem__s7'%c*!+GU!;!;g'2 }V}rrE)r6r7r8r9rr<r=r>s@rrrs rrc^\rSrSrSrSSSS\R S4U4SjjrS\S \4S jr S\S \4S jr S \S \4S jr Sr U=r $)XIELUActivationz Applies the xIELU activation function introduced in https://arxiv.org/abs/2411.13010 If the user has installed the nickjbrowning/XIELU wheel, we import xIELU CUDA Otherwise, we emit a single warning and use xIELU Python g?r#gưFc <>[T U]5 [R"[R "[R "[R"XS955RS55Ul [R"[R "[R "[R"X#- US955RS55Ul URS[R"X5S95 URS[R"XES95 X`l [U5Ul[U5UlSUlSSKn[R$R&R)5UlSnSSKJn U "UR.5UlUS- n[4R7U5 g![2a$n USU S 3- nUR.UlSn A N?Sn A ff=f![2a#n [4R7S U S 35 Sn A gSn A ff=f) N)dtyperbetaepszUsing experimental xIELU CUDA.)allow_in_graphz& Enabled torch._dynamo for xIELU CUDA.z+ Could not enable torch._dynamo for xIELU (z*) - this may result in slower performance.z CUDA-fused xIELU not available (u) – falling back to a Python version. For CUDA xIELU (experimental), `pip install git+https://github.com/nickjbrowning/XIELU`)rrr Parameterr(logexpm1tensor unsqueezealpha_palpha_nregister_bufferwith_vector_loadsru _beta_scalar _eps_scalar_xielu_cuda_obj xielu.opsclassesxieluXIELUtorch.compilerr _xielu_cuda_xielu_cuda_fn Exceptionlogger warning_once) r alpha_p_init alpha_n_initrrrrrmsgrerrrs rrXIELUActivation.__init__s ||EIIekk%,,|:a.b$c$m$mno$pq || IIekk%,,|/B%"PQ R \ \]^ _   VU\\$%DE UELL$BC!2!$K :#  #(==#6#6#<#<#>D 2C 79&4T5E5E&F#??    $ 7DSEIstt&*&6&6## 7    23%8jj   sB3G."F='G.= G+G&!G.&G++G.. H8HHrqr c [RRUR5nUR[RRUR 5-n[ R"US:X!-U-URU--[ R"[ R"XR55U- U-URU--5$)Nr) rrrrrrr(whererrkr)rrqrrs r _xielu_pythonXIELUActivation._xielu_pythons--((6))bmm44T\\BB{{ E K!Odii!m + [[1hh/ 01 4 ?$))a- O  rcURnUR5S:a'URS5nUR5S:aM'UR5S:a"URSSUR S55nX!R:wa![ R SUUR5 URRUURRUR5URRUR5URURUR5nURU5$)zDFirewall function to prevent torch.compile from seeing .item() callsrrrz_Warning: xIELU input tensor expects 3 dimensions but got (shape: %s). Reshaping to (shape: %s).)shapedimrviewsizerrrr2rtorrrrr)rrqoriginal_shaperesults rrXIELUActivation._xielu_cuda"seegk AAeegk 557Q;r1affRj)A WW $   q  %%-- LLOOAGG $ LLOOAGG $        " " {{>**rrcURbFUR(a5[5(dURU5$[R S5 UR U5$)Nz:torch._dynamo is compiling, using Python version of xIELU.)ris_cudar rrrrr-s rr2XIELUActivation.forward;sN    + +--**511##$`a!!%((r)rrrrrrr)r6r7r8r9r:r(bfloat16rrrrr2r<r=r>s@rrrsd  nn(T v & +V++2)V)))rrrgelu_10i )rkrl gelu_fastgelu_new gelu_pythonrKTgelu_pytorch_tanhgelu_python_tanhr gelu_accurate hardswishlaplace leaky_relulinearr quick_gelurrelu2relu6)rfrY sqrtsoftplusswishrprelurc U[;a [U$[SUS[[R5535e)Nz function z not found in ACT2FN mapping )ACT2FNKeyErrorlistkeys)activation_strings rget_activationrasBF"'((#4"55RSWX^XcXcXeSfRghiirrY)5rr) collectionsrr(rrintegrations.hub_kernelsrutilsrutils.import_utilsr get_loggerr6rModuler PytorchGELUTanhrArIrVr]rcrirwrrrrrrr Hardswish LeakyReLUReLUReLU6SigmoidrTTanhPReLUACT2CLSrrrrrrrrrYr linear_actrErrrsJ # A8   H %Z(ryy)0Y'w w(wV$RYY%,V$ )RYY )% )Z(ii)i[)4"))4*47BII72 tRYY t RYY"ryy . .BII4RYY4kZ)biiZ)z  N  %s2'>? # !   N%6$=>     $:D#AB +    ",,    N %  BGG   "! " RXX# $zz * WW GG XX 1 4  !j]+ * %f ; ' "#67 L ) ff H % r