Zj(SSKJr SSKrSSKJr SSKJr SSKJrJ r J r J r J r SSK Jr SS KJr \R""\5rS r"S S \5r"S S\ 5rSrSSjr"SS\5r"SS\ 5r"SS\ 5r"SS\ 5r/SQrg))OptionalN)logging)LlamaAttentionLlamaForCausalLMLlamaForSequenceClassificationLlamaForTokenClassificationLlamaRotaryEmbedding)Phi3MLP) GlmConfigzTHUDM/glm-4-9bc\rSrSrSrg)GlmMLP%N__name__ __module__ __qualname____firstlineno____static_attributes__rt/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/glm/modular_glm.pyrr%rrc ^\rSrSr\S S\S-S\SS\S-S\S\ 44S jj5r S r g) GlmRotaryEmbedding)Nconfigdevicez torch.deviceseq_lenreturnz torch.Tensorc jURSnURRSS5n[USS5=(d URUR-n[ XT-5nSnSU[ R"SUS[ RS9RU[ RS 9U- -- nX4$) aH Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetapartial_rotary_factorg?head_dimNrr)dtype)r r') rope_parametersgetgetattr hidden_sizenum_attention_headsinttorcharangeint64tofloat) rr r!baser%r&dimattention_factorinv_freqs rcompute_default_rope_parameters2GlmRotaryEmbedding.compute_default_rope_parameters*s&%%l3 & 6 6 : :;RTW X6:t4h8J8JfNhNh8h(23 U\\!S!5;;?BB&X]XcXcBdgjj k ))rr)NNN) rrrr staticmethodrrr-tupler2r7rrrrrr)sY#'+/"*D *(*t* ~u$ % **rrcxUSSSS24nUSSSS24n[R"U*U4SS9RS5$) z*Rotates half the hidden dims of the input..rNrr r4)r.stackflatten)xx1x2s r rotate_halfrDKsJ 319B 319B ;;Ryb ) 1 1" 55rcURU5nURU5nUSSURSS-24RSSS9nUSSURSS-24RSSS9nURSnUSSU24USUS24pvUSSU24USUS24pXb-[U5U--n X-[U5U--n [R "X/SS9n [R "X/SS9n X4$)aIApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. .Nr<rr=) unsqueezeshaperepeat_interleaverDr.cat) qkcossin unsqueeze_dim rotary_dimq_rotq_passk_rotk_passq_embedk_embeds rapply_rotary_pos_embrVRs6$ -- &C -- &C c'SYYr]a''' ( : :1" : EC c'SYYr]a''' ( : :1" : EC2Jc;J;&'3 +;)<6c;J;&'3 +;)<6{{51C78G{{51C78Gii)r2Gii)r2G  rc>^\rSrSrSS\S\S-4U4SjjjrSrU=r$) GlmAttentionzNr layer_idxc>[TU]X5 [R"URUR -UR SS9Ulg)NF)bias)super__init__nnLinearr,r&r+o_proj)selfrrZ __class__s rr^GlmAttention.__init__{s: +ii : :T]] JFL^L^ejk r)ra)N) rrrrrr-r^r __classcell__)rcs@rrXrXzs#lylS4ZllrrXc\rSrSrSrg)GlmForCausalLMrNrrrrrgrgrrrgc\rSrSrSrg)GlmForSequenceClassificationrNrrrrrjrjrrrjc\rSrSrSrg)GlmForTokenClassificationrNrrrrrmrmrrrm)GlmPreTrainedModelGlmModelrgrjrm)r )typingrr.torch.nnr_utilsrllama.modeling_llamarrr r r phi3.modeling_phi3r configuration_glmr get_loggerrlogger_CHECKPOINT_FOR_DOCrrrDrVrXrgrjrm__all__rrrr{s )(   H %& W *-*D6%Pl>l  %  #A  ;  r