Zj%SSKrSSKrSSKJr SSKJr SSKJrJrJ r SSK J r J r \ R"\5r\ "5(aSSKr\(aSSKJr SrS!S \S S \S S \S-S\S-S\S\44 SjjrS"S \S S \S S \S-S\S-S\S\S\44 SjjrS!S \S S \S S \S-S\S-S\S\44 SjjrS#S S S \S S \S-S\S-S\S\44 SjjrS#S S S \S S \S-S\S-S\S\44 SjjrS#S S S \S S \S-S\S-S\S\44 Sjjr\\\\\\S.r\\\S\S\444\ S'"SS\ 5r!"SS5r"S$S \"S\#S-4S jjr$g)%N)Callablewraps) TYPE_CHECKINGOptional TypedDict)is_torch_availablelogging)PreTrainedConfigcP^^^SSjmSSjm[T5SUUU4Sjj5nU$)aD Decorator function to update the RoPE parameters in the forward pass, if the model is using a dynamic RoPE (i.e. a RoPE implementation that may recompute its frequencies in the forward pass). Args: rope_forward (Callable): The forward pass of the RoPE implementation. Returns: The decorated forward pass. cB[R"U5S-nUc4URnURnSnURR SnO>URUn[ XS35nUS3nURR USnXH:aX[XS35(d![Un U "URUUS-US9upURUS 3W S S 9 [XS 3U 5 gURU5nURUS 3US S 9 [XS 3U5 g)zbLongrope uses long factor if sequence is larger than original pretraining length, short otherwise.r N original_max_position_embeddings_original_inv_freq__long_inv_freqseq_len layer_typeinv_freqF persistent long_inv_freqoriginal_inv_freq) torchmax rope_typerconfigrope_parametersgetattrhasattrROPE_INIT_FUNCTIONSregister_buffersetattrto) self position_idsdevicerrrrprefixr rope_init_fnrrs q/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/modeling_rope_utils.pylongrope_frequency_update6dynamic_rope_update..longrope_frequency_update/sO))L)A-  I $ 6 6 F/3{{/J/JKm/n ,z2I ' ??29= #/KK.dynamic_frequency_updateRsi ))L)A-  I!%!8!8  $ 6 6 Fz2I!(=P/QSWSjSj!k  ' W>W X 4b .r/c>Uc UROURUnUbSU0O0nSU;aT"X4SUR0UD6 OUS:XaT"X4SUR0UD6 T"XU40UD6$)Nrdynamicr)longrope)rr)) r'xr(rrkwargsr5r- rope_forwards r,wrapper$dynamic_rope_update..wrapperxs{&0&8DNNdnnZ>X /9/E, +2  ! $T S SF S * $ %d T TV TD\ W$v'E'EE&ST*U[^hibi[jk kDdu||AsAU[[ILLTZbgbmbmLnqttuvH  %%r/c^UR5 UbURUO URnUSnURSS5n[USURUR -5n[ Xv-5nUSn URS5n URS5n URS5n US n U cURU - n SS jnU c1U (a"U (a[U"X5U"X5- 5n OU"U 5n URS 5=(d S nURS5=(d S nSmU4SjnSnU[R"SUS5RU[RS9U- -nSU- nSU U-- nURRSS5nU"UUXU U5unnS U"UUUS-5RU[RS9- nUS U- -UU--nUU 4$)a Computes the inverse frequencies with NTK scaling. Please refer to the [original paper](https://huggingface.co/papers/2309.00071) Args: config ([`~transformers."PreTrainedConfig"`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`, *optional*): The base wavelength from which the inverse frequencies will be derived. Defaults to `config.default_theta` if omitted. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. * max_position_embeddings (`int`): The maximum length of the positional embeddings. * rope_parameters (`dict[str, float | int]`): The standard RoPE scaling parameters, from which the following keys will be accessed: * `attention_factor` (`float`, *optional*): The scaling factor to be applied to the computed cos/sin. If None, the value is inferred from `factor`, `mscale`, and `mscale_all_dim` as available. * `beta_fast` (`float`, *optional*, defaults to 32): Parameter to set the boundary for extrapolation (only) in the linear ramp function. * `beta_slow` (`float`, *optional*, defaults to 1): Parameter to set the boundary for interpolation (only) in the linear ramp function. * `factor` (`float`, *optional*): The scaling factor applied when interpolating the position IDs to extend the possible context length. Additionally, if `attention_factor` is None, the log of this value is used to compute a value for `attention_factor`, possibly in conjunciton with `mscale` and `mscale_all_dim`, if provided. * `mscale` (`float`, *optional*): If `attention_factor` is None and both `mscale` and `mscale_all_dim` are provided, `mscale` acts scalar augmenting `log(factor)` when computing the numerator for the inferred value of `attention_factor`. If not provided, `attention_factor` will be calculated based on `factor` only. * `mscale_all_dim` (`float`, *optional*): If `attention_factor` is None and both `mscale` and `mscale_all_dim` are provided, `mscale_all_dim` acts scalar augmenting `log(factor)` when computing the denominator for the inferred value of `attention_factor`. If not provided, `attention_factor` will be calculated based on `factor` only. * `original_max_position_embeddings` (`int`): The original max position embeddings used during pretraining. * `truncate` (`bool`, *optional*): Whether to truncate the correction range. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*, defaults to 1.0): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. rDrErFrGrCrWmscalemscale_all_dimrr cNUS::agSU-[R"U5-S-$)Nr rFg?)mathlog)scalerls r, get_mscale,_compute_yarn_parameters..get_mscales( A:V|dhhuo-33r/ beta_fast beta_slowcU[R"X0S-[R-- 5-S[R"U5-- $)zPInverse dimension formula to find the dimension based on the number of rotationsrH)rorppi) num_rotationsrVrUres r,find_correction_dim5_compute_yarn_parameters..find_correction_dims@dhh6!:Kdgg:UVWW\]`d`h`him`n\noor/c>T"XX45nT"XX45nU(a,[R"U5n[R"U5n[US5[ XrS- 54$)z.Find dimension range bounds based on rotationsrr )rofloorceilrmin) low_rothigh_rotrVrUretruncatelowhighrzs r,find_correction_range7_compute_yarn_parameters..find_correction_rangesR!'N"8$P **S/C99T?D3{CAg...r/cX:XaUS- n[R"U[RS9U- X- - n[R"USS5nU$)NgMbP?rIrr )rrQr^clamp)rrrV linear_func ramp_funcs r,linear_ramp_factor4_compute_yarn_parameters..linear_ramp_factorsH : 5LC||Cu}}=C R KK Q2 r/rrHrKrT)r ) rLr rMr!rNrOrPrerSrrQr&)rr)rrrTrUrErGrVrCrWrlrmrrrrtrvrr pos_freqsinv_freq_extrapolationinv_freq_interpolationrrrinv_freq_extrapolation_factorrrzs @r,_compute_yarn_parametersrGs/t ""$AKAW611*=]c]s]s  -D0445LcRvz6+=+=A[A[+[\H h. /C !( +F+//0BC ! % %h /F)--.>?N';<^'_$ ~//2RR4  n$Z%?*VBd%de )&1 %((5;I$((5:Ip/aa03363UX[[\I 9_ FY$67%%))*d;H%iCGgiqrIC%&(:3cQh(O(R(RZ`hmhshs(R(t$t!!&C"CD #@ @ A  % %%r/cHUR5 UbURUO URnUSnURSS5n[USURUR -5n[ Xv-5nUSn USn URS5n URS5n US n U cURU - n U cMU S::aSn OD[R"S [R"U 5[R"U 5- -5n U(a*X-:a%[R"U [RUS 9nO$[R"U [RUS 9n[R"S US [RUS 9R!5U- nSXU--- nUU 4$)ay Computes the inverse frequencies with LongRoPE scaling. Please refer to the [original implementation](https://github.com/microsoft/LongRoPE) Args: config ([`~transformers."PreTrainedConfig"`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`, *optional*): The base wavelength from which the inverse frequencies will be derived. Defaults to `config.default_theta` if omitted. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. * max_position_embeddings (`int`): The maximum length of the positional embeddings. * original_max_position_embeddings (`int`, *optional*): The original max position embeddings used during pretraining. If not provided, defaults to `max_position_embeddings`. * rope_parameters (`dict[str, float]`): The standard RoPE scaling parameters, from which the following keys will be accessed: * `attention_factor` (`float`, *optional*): The scaling factor to be applied on the attention computation. If unspecified, it defaults to value recommended by the implementation, inferred from the value of `factor`. * `factor` (`float`, *optional*): The scaling factor to apply to the RoPE embeddings. If both `max_position_embeddings` and `original_max_position_embeddings` are provided, this value will be overridden s the ratio between those values. * `long_factor` (`float`, *optional*): The scale factor applied when computing the inverse frequencies if `seq_len` is provided and greater than `original_max_position_embeddings`. * `short_factor` (`float`, *optional*): The scale factor applied when computing the inverse frequencies if `seq_len` is None or less-than-or-equal-to `original_max_position_embeddings`. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*, defaults to 1.0): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. rDrErFrG long_factor short_factorrCrWrr r[rrH)rLr rMr!rNrOrPrerosqrtrprrir^rQrRrS)rr)rrrTrUrErGrVrrrCrWr ext_factorsinv_freq_shapers r,_compute_longrope_parametersrsd ""$AKAW611*=]c]s]s  -D0445LcRvz6+=+=A[A[+[\H h. /C&}5K'7L ! % %h /F+//0BC';<^'_$ ~//2RR S=" #yyTXXf-=Ii@j-j)jk 7=ll;emmFS ll?*+MNO&8':$''kH$G[[+;!;X=NPXYN$w.@EUEghM]*n