ZjHSSKrSSKJr SSKJr SSKJr SSKJr SSK J r SSK J r J r S S KJrJrJrJr S S KJr S S KJr S SKJr \ R2"\5rSr"SS\5r"SS\5r"SS\5r"SS\5r "SS\5r!"SS\5r"/SQr#g)N)Cache)FlashAttentionKwargs)GradientCheckpointingLayer)CausalLMOutputWithPast)Unpack)TransformersKwargslogging) GlmAttentionGlmForCausalLMGlmForSequenceClassificationGlmForTokenClassification)Phi3MLP) Glm4Config) Glm4RMSNormzTHUDM/GLM-4-9B-0414c\rSrSrSrg)Glm4MLP#N__name__ __module__ __qualname____firstlineno____static_attributes__rv/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/glm4/modular_glm4.pyrr#rrcT^\rSrSrS\S\4U4SjjrSS\RS\RS-S\RS-S \ S-S \ S-S \ \R\R4S-S \ \S \ \R\ \R\R4S-44SjjrSrU=r$)Glm4DecoderLayer'config layer_idxc>[TU]5 URUl[XS9Ul[ U5Ul[URURS9Ul [URURS9Ul [URURS9Ul [URURS9Ul g)N)r$r%)eps) super__init__ hidden_size Glm4Attention self_attnrmlpr rms_norm_epsinput_layernormpost_attention_layernormpost_self_attn_layernormpost_mlp_layernorm)selfr$r% __class__s rr)Glm4DecoderLayer.__init__(s !--&fJ6?*6+=+=6CVCVW(3F4F4FFL_L_(`%(3F4F4FFL_L_(`%"-f.@.@fFYFY"ZrN hidden_statesattention_mask position_idspast_key_values use_cacheposition_embeddingskwargsreturnc UnURU5nUR"SUUUUUUS.UD6upURU5nX-nUnURU5nUR U5nUR U5nX-nU$)N)r6r7r8r9r:r;r)r/r,r1r0r-r2) r3r6r7r8r9r:r;r<residual_s rforwardGlm4DecoderLayer.forward3s!,,]; >> ')%+ 3    55mD  0  55mD / // >  0 r)r*r/r-r0r2r1r,)NNNFN)rrrrrintr)torchTensor LongTensorrbooltuplerr FloatTensorrAr __classcell__r4s@rr"r"'s [z [c [/304(,!&HL|| t+&&-    $; #5<<#=>E-. u  %(9(95;L;L(L"MPT"TT Urr"c\rSrSrSrg)r+UrNrrrrr+r+Ur rr+c@^\rSrSrS\\S\\-4U4SjjrSr U=r $)Glm4ForCausalLMY super_kwargsr=c $>[TU]"S0UD6$)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, Glm4ForCausalLM >>> model = Glm4ForCausalLM.from_pretrained("THUDM/GLM-4-9B-0414") >>> tokenizer = AutoTokenizer.from_pretrained("THUDM/GLM-4-9B-0414") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```r)r(rA)r3rQr4s rrAGlm4ForCausalLM.forwardZs4w...rr) rrrrrr rHrrArrJrKs@rrOrOYs+/12/ ' '//rrOc\rSrSrSrg)Glm4ForSequenceClassificationwrNrrrrrUrUwr rrUc\rSrSrSrg)Glm4ForTokenClassification{rNrrrrrXrX{r rrX)Glm4PreTrainedModel Glm4ModelrOrUrX)$rD cache_utilsrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrprocessing_utilsrutilsr r glm.modeling_glmr r rrphi3.modeling_phi3rconfiguration_glm4r modeling_glm4r get_loggerrlogger_CHECKPOINT_FOR_DOCrr"r+rOrUrX__all__rrrrjs  B96&0tt(*&   H %+ g +1+\ L /n/< $@  !:  r