Zj}dSSKrSSKJr SSKJr SSKJr SSKrSSKJr SSK J r SSK J r SS KJrJr SS KJr SS KJrJrJr SS KJrJr SS KJr SSKJrJrJrJ r J!r! SSK"J#r#J$r$ SSK%J&r&J'r' SSK(J)r) SSK*J+r+J,r,J-r- SSK.J/r/J0r0 SSK1J2r2J3r3 SSK4J5r5J6r6 Sr7"SS\Rp5r9"SS\Rp5r:Sr;\"S5S_Sj5r<S`S\RpS \RzS!\RzS"\RzS#\RzS-S$\>S-S%\>S&\)\+4S'jjr?\"\<5"S(S)\Rp55r@"S*S+\Rp5rA"S,S-\Rp5rBS.rC"S/S0\Rp5rD"S1S2\Rp5rE"S3S4\5rF"S5S6\Rp5rG"S7S8\Rp5rH\,"S9S:\'55rI"S;S<\I5rJ"S=S>\Rp5rK"S?S@\Rp5rL"SASB\Rp5rM\\,"SCSD\!555rN"SESF\Rp5rO"SGSH\Rp5rP\"SI5"SJSK\Rp55rQ"SLSM\Rp5rR"SNSO\Rp5rSSP\RzSQ\TSR\Rz4SSjrU\"\<5"STSU\Rp55rV"SVSW\5rW\,"SXSY\'55rX"SZS[\X5rY"S\S]\X\5rZ/S^Qr[g)aN)Callable) dataclass)Optional)nn)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hubuse_kernel_func_from_hubuse_kernelized_func)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer)"BaseModelOutputWithCrossAttentionsBaseModelOutputWithPast,BaseModelOutputWithPoolingAndCrossAttentionsCausalLMOutputWithPast ModelOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuple)maybe_autocastmerge_with_config_defaults)OutputRecordercapture_outputs) EvollaConfig SaProtConfigcURU5R5n[R"USS9R U5U-nUR 5U-$)z Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. Args: x: torch.Tensor x: Returns: torch.Tensor r$dim)neinttorchcumsumtype_aslong) input_ids padding_idxmaskincremental_indicess {/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/evolla/modeling_evolla.py"create_position_ids_from_input_idsr54sP << $ ( ( *D,,t3;;DADH  # # % 33cD^\rSrSrSrU4SjrSSjrSrSrU=r $)EvollaSaProtEmbeddingsDzN Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. c>[TU]5 [R"URUR UR S9UlUR(a/[R"UR URS9Ul OSUl [R"UR5Ul[USS5UlUR#S[$R&"UR(5R+S5SS9 UR UlUR S:Xa9[R"UR(UR UR,S9UlUR0UlUR2UlSUlg) N)r1epsposition_embedding_typeabsolute position_ids)r$F persistent)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsemb_layer_norm_before LayerNormlayer_norm_eps layer_normDropouthidden_dropout_probdropoutgetattrr=register_bufferr,arangemax_position_embeddingsexpandr1position_embeddings token_dropout mask_token_idr?selfconfig __class__s r4rDEvollaSaProtEmbeddings.__init__Is2 !||F,=,=v?Q?Q_e_r_rs  ' ' ll6+=+=6CXCXYDO"DOzz&"<"<= '.v7PR\']$  ELL)G)GHOOPWXej  "..  ' ': 5')||..0B0BPTP`P`(D $$11#11 r6cUc*Ub[XR5nOURU5nUcURU5nUnUR(aUbUR XR :HRS5S5nSnUbURS5OURSnXR :HRS5R5U- nUSU- -SU- SS2SS4- RUR5nURS:XaURU5n XY-nURbURU5nUb,XRRS5-RUR5nU$)Nr@gQ?r$r>)r5r1&create_position_ids_from_inputs_embedsrIrW masked_fillrX unsqueezesumshapefloattodtyper=rVrM) rZr0attention_maskr? inputs_embeds embeddingsmask_ratio_train src_lengthsmask_ratio_observedrVs r4forwardEvollaSaProtEmbeddings.forwardbs  $A)M]M]^ #JJ=Y   00;M#    )"7#//>P>P1P0[0[\^0_adeJ) 4B4N.,,R0T]TcTcdeTfK#,0B0B#B"G"G"K"Q"Q"SVa"a $,<(<=EXAXZ[]acgZg@hhll  J  ' ': 5"&":":<"H #9J ?? &4J  %$'?'?'CCGG HXHXYJr6cUR5SSnUSn[R"URS-X0R-S-[RUR S9nUR S5RU5$)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr@r$)rgdevicer)sizer,rSr1r/rqrbrU)rZri input_shapesequence_lengthr?s r4r`=EvollaSaProtEmbeddings.create_position_ids_from_inputs_embedss~$((*3B/ %a.||   q /4D4D"Dq"HPUPZPZcpcwcw %%a(// <[TU]5 Xl0UlUR URU5up4UR SU5 [ USU5 g)Nrattention_scaling)rCrDr[ rope_typecompute_default_rope_parametersrRsetattr)rZr[rq curr_inv_freqcurr_attention_scalingr\s r4rD$EvollaSaProtRotaryEmbedding.__init__sU  040T0TUYU`U`bh0i-  Z7)+ABr6rqztorch.device | Noneseq_lenreturn torch.Tensorc URn[USS5=(d URUR-nSnSU[R "SUS[R S9RU[RS9U- -- nXe4$)aI Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). head_dimN?rrgrqrg) rope_thetarQrGnum_attention_headsr,rSint64rfrer[rqrbaser)attention_factorrs r4r;EvollaSaProtRotaryEmbedding.compute_default_rope_parameterss(  fj$/c63E3EIcIc3c U\\!S!5;;?BB&X]XcXcBdgjj k ))r6c@[US5n[US5nUSSS2S4R5RURSSS5R UR 5nUSS2SSS24R5n[ UR R[5(a0UR RS:waUR ROSn[USS 9 UR5UR5-RSS 5n [R"X4SS 9n U R5U-n U R5U-n SSS5 W R URS 9W R URS 94$!,(df  N@=f) Nrrrr@r$mpscpuF device_typeenabledrr(r)rQrerUrdrfrq isinstancetypestrr transposer,catcossinrg) rZxr? layer_typerrinv_freq_expandedposition_ids_expandedrfreqsembrrs r4rn#EvollaSaProtRotaryEmbedding.forwardsZ4,#D*=>$T1d]399;BB > @'1!((--'E'E!((--[`J`ahhmmfk  U C&,,.1F1L1L1NNYYZ[]^_E))UN3C'')//C'')//C D vvAGGv$cff177f&;;; D Cs 'A.F F)r[rNNNN)rwrxryrzr{r,Tensor__annotations__r&rD staticmethodr+tuplererno_gradrrnr|r}r~s@r4rrs llC|CC&*(,"*t#*%*t* ~u$ % **< ]]_<