Zj SSKrSSKJr SSKJr SSKJr SSKrSSKJ r SSK J s J r SSK Jr SSKJr SSKJr SS KJrJr SS KJr SS KJr SS KJrJr SS KJr SSK J!r! SSK"J#r#J$r$J%r%J&r&J'r' SSK(J)r)J*r* SSK+J,r,J-r- SSK.J/r/ SSK0J1r1J2r2J3r3J4r4J5r5 SSK6J7r7J8r8 SSK9J:r: SSK;J\T55rU"S?S@\T\5rV\2"SASB9\"SCSD\'555rW"SESF\RR5rX"SGSH\ R5rYSIrZ"SJSK\ R5r[SL\RS/\R4SMjr\S/\RS0\RSL\RS)\K\R\R44SNjr]"SOSP\ R5r^"SQSR\ R5r_"SSST\!5r`"SUSV\ R5ra"SWSX\ R5rb"SYSZ\ R5rc"S[S\\T5rd"S]S^\T\5re/S_Qrfg)aN)Callable) dataclass)Optional)Llama4VisionConfig)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hub)create_causal_maskcreate_chunked_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPastBaseModelOutputWithPoolingCausalLMOutputWithPast ModelOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleloggingtorch_compilable_check)maybe_autocastmerge_with_config_defaults)capture_outputs) Llama4ConfigLlama4TextConfigcj^\rSrSrS\4U4SjjrS\RS\R4SjrSr U=r $)Llama4TextExperts8configc>[TU]5 URUlURUlUR UlURUl[R"[R"URUR SUR -55Ul [R"[R"URUR UR 455Ul [URUlgN)super__init__num_local_experts num_expertsintermediate_size hidden_size expert_dimnn Parametertorchzeros gate_up_projempty down_projr hidden_actact_fnselfr* __class__s {/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/llama4/modeling_llama4.pyr/Llama4TextExperts.__init__9s !33!'!9!9!--00LLT5E5EtGWGWYZ]a]l]lYl)mnekk43C3CT__VZVfVf2g&hiV../ hidden_statesreturncnURURRSSUR5n[R "XR5nUR SSS9up4[R "X@RU5-UR5nURSUR5nU$)a This should really not be run on a single machine, as we are reaching compute bound: - the inputs are expected to be "sorted" per expert already. - the weights are viewed with another dim, to match num_expert, 1, shape * num_tokens, shape Args: hidden_states (torch.Tensor): (batch_size * token_num, hidden_size) selected_experts (torch.Tensor): (batch_size * token_num, top_k) routing_weights (torch.Tensor): (batch_size * token_num, top_k) Returns: torch.Tensor rr-dim) viewr9shaper3r7bmmchunkr=r;)r?rDgate_upgateup next_statess rAforwardLlama4TextExperts.forwardCs&**4+<+<+B+B1+Er4K[K[\ ))M+<+<====+iikk$&7!7$..I !&&r4+;+;< rC)r=r;r4r9r3r2r1) __name__ __module__ __qualname____firstlineno__r&r/r7TensorrR__static_attributes__ __classcell__r@s@rAr(r(8s00/0U\\ellrCr(c2^\rSrSrSU4SjjrSrSrU=r$) Llama4TextMLPYcX>[TU]5 Uc URnXl[R "UR USS9Ul[R "UR USS9Ul[R "X!R SS9Ul [URUl gNFbias) r.r/r2r*r5Linearr3 gate_projup_projr;r r< activation_fn)r?r*r2r@s rAr/Llama4TextMLP.__init__Zs   $ & 8 8  6#5#57HuUyy!3!35FUS #46H6HuU#F$5$56rCcURURU55URU5-nURU5$N)rfrdrer;)r?xr;s rArRLlama4TextMLP.forwardfs7&&t~~a'89DLLOK ~~i((rC)rfr*r;rdrerirTrUrVrWr/rRrYrZr[s@rAr]r]Ys 7))rCr]cF^\rSrSrSS\4U4SjjjrSrSrSrSr U=r $) Llama4TextL2Normkepsc.>[TU]5 Xlgri)r.r/rp)r?rpr@s rAr/Llama4TextL2Norm.__init__ls rCcU[R"URS5RSSS9UR-5-$Nr-rGT)keepdimr7rsqrtpowmeanrpr?rjs rA_normLlama4TextL2Norm._normp45;;quuQx}}R}>IJJJrCc^URUR55RU5$ri)r{floattype_asrzs rArRLlama4TextL2Norm.forwardss"zz!'')$,,Q//rCc SUR3$)Nzeps=rpr?s rA extra_reprLlama4TextL2Norm.extra_reprvsdhhZ  rCr)gư>) rTrUrVrWrr/r{rRrrYrZr[s@rArnrnks)EK0!!rCrnc>^\rSrSrSU4SjjrSrSrSrSrU=r $)Llama4TextRMSNormzc>[TU]5 X l[R"[ R "U55Ulg)z, Llama4RMSNorm is equivalent to T5LayerNorm N)r.r/rpr5r6r7onesweight)r?r3rpr@s rAr/Llama4TextRMSNorm.__init__{s. ll5::k#:; rCcU[R"URS5RSSS9UR-5-$rtrvrzs rAr{Llama4TextRMSNorm._normr}rCczURUR55RU5nX R-$ri)r{rrr)r?rjoutputs rArRLlama4TextRMSNorm.forwards.AGGI&..q1 ##rCc^[URR5SUR3$)Nz, eps=)tuplerrKrprs rArLlama4TextRMSNorm.extra_reprs' ))*+6$((<) rTrUrVrWr/r{rRrrYrZr[s@rArrzs<K$==rCrc4^\rSrSrU4SjrU4SjrSrU=r$) Llama4Routerc>[TU]URURSS9 URUlUR Ulgr`)r.r/r3r0r1num_experts_per_toktop_kr>s rAr/Llama4Router.__init__s> ++V-E-EER!33// rCcj>[TU]U5n[R"X RSS9up4[R "U[ S55RSXC5n[RRRUR 55RUR5nXR4$)Nr$rHz-inf) r.rRr7topkr full_likerscatter_r5 functionalsigmoidtodtype)r?rD router_logitsrouter_top_valuerouter_indices router_scoresr@s rArRLlama4Router.forwards 6 +0::mZZUV+W( uV}ENNqR`s ++33M4G4G4IJMMmNaNab ++rC)r1rrlr[s@rArrs0 ,,rCr Llama4TextMoec.^\rSrSrU4SjrSrSrU=r$)rc>[TU]5 URUlURUlUR Ul[U5Ul [U5Ul [U5Ul gri)r.r/rrr3 hidden_dimr0r1r(expertsrrouterr] shared_expertr>s rAr/Llama4TextMoe.__init__s[ //  ,,!33(0 "6* *62rCcURSUR5nURU5up#URURSS5nXBR SS5RSS5-nUR U5nURU5nURURURSSURS5RSS95 Xc4$)NrGr$rrH) reshaperrrepeatrK transposerradd_sum)r?rDrr routed_in routed_outouts rArRLlama4TextMoe.forwards%--b$//B '+{{='A$ !(()<)[TU]5 URUlURUlXlUR R SUlURnURS:wa[URnU"UR U5uo@l URSUSS9 URSUR5SS9 g)N rope_typedefaultrF persistentoriginal_inv_freq) r.r/max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr*rope_parametersrcompute_default_rope_parametersrattention_scalingregister_bufferclone)r?r*device rope_init_fnrr@s rAr/"Llama4TextRotaryEmbedding.__init__s "("@"@$*$B$B! 44[A!%!E!E >>Y &.t~~>L+7 V+L(( ZeD 0(..2BuUrCrz torch.deviceseq_lenrEz torch.Tensorc URSn[USS5=(d URUR-nSnSU[R "SUS[R S9RU[RS9U- -- nXe4$) aH Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetahead_dimN?rr-r)rr) rgetattrr3num_attention_headsr7arangeint64rr)r*rrbaserIattention_factorrs rAr9Llama4TextRotaryEmbedding.compute_default_rope_parameterss&%%l3fj$/c63E3EIcIc3c U\\!S!5;;?BB&X]XcXcBdgjj k ))rCcURSSS2S4R5RURSSS5nUSS2SSS24R5n[ UR R [5(a0UR R S:waUR R OSn[USS9 URUR 5U-RSS5n[R"[R"U5U5nXpR-nSSS5 U$!,(df  W$=f) NrrGr$mpscpuF) device_typeenabledr-)rrexpandrK isinstancertypestrr!rrr7polar ones_liker)r?rj position_idsinv_freq_expandedposition_ids_expandedrfreqs freqs_ciss rArR!Llama4TextRotaryEmbedding.forwards!MM$4-8>>@GG HZHZ[\H]_acde ,QaZ 8 > > @'1!((--'E'E!((--[`J`ahhmmfk  U C&))!((36KKVVWXZ[\E EOOE$:EBI!$:$::ID  D C s A(D22 E)rr*rrrri)NNN)rTrUrVrWr7rX__annotations__r&r/ staticmethodrintrrrno_gradrrRrYrZr[s@rArrsllV/VV *.+/"* 4'*(*t* ~u$ % **< ]]_  rCrxqxkrrEc *[R"UR5R"/URSSQSPSP765n[R"UR5R"/URSSQSPSP765n[R "X2SS2SS2SSS24-5R S5n[R "XBSS2SS2SSS24-5R S5nURU5URU54$)NrGr-r)r7view_as_complexrrrK view_as_realflattenr)rrrxq_xk_xq_outxk_outs rAapply_rotary_embrs    2 2 IBHHSbM I2 Iq I JC    2 2 IBHHSbM I2 Iq I JC   1dA &> > ? G G JF   1dA &> > ? G G JF >>" v~~b1 11rCrDn_repcURup#pEUS:XaU$USS2SS2SSS2SS24RX#XU5nURX#U-XE5$)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r$N)rKrr)rDrbatchnum_key_value_headsslenrs rA repeat_kvrs_ 2?1D1D.E z!!Qa"23::5W\dlmM  e(CT TTrCmodulequerykeyvalueattention_maskscalingdropoutc [X R5n[X0R5n [R"XR SS55U-n UbX-n [ R RU SS9n [ R RXURS9n [R"X5n U R SS5R5n X4$)Nr-rrGrHptrainingr$) rnum_key_value_groupsr7matmulrr5rsoftmaxr r contiguous rrrrr r r kwargs key_states value_states attn_weights attn_outputs rAeager_attention_forwardrs3 ; ;L==((6??([L,,|:K''1-88:K  $$rCc [X R5n[X0R5n [R"XR SS55UR S--n UbX-n [ RRU SS9n [ RRXURS9n [R"X5n U R SS5R5n X4$)Nr-rrGrHr r$) rrr7rrrr5rrr rrrs rAvision_eager_attention_forwardr(s3 ; ;L==((6??([L,,|:K''1-88:K  $$rCc"^\rSrSrSrS\4U4SjjrSS\RS\ \R\R4S\RS-S \ S-S \ \ S \ \R\RS-\ \RS-44 S jjr S rU=r$)Llama4TextAttentioniAz=Multi-headed attention from 'Attention Is All You Need' paperr*c>[TU]5 XlX l[ USUR UR -5UlUR UlUR UR-Ul URUlURS-Ul URUl URUl URUl URUlSUlUR UUl[$R&"UR UR UR-UR(S9Ul[$R&"UR URUR-UR(S9Ul[$R&"UR URUR-UR(S9Ul[$R&"UR UR-UR UR(S9UlURR2(a-UR"(aR65Ulggg)NrrTra)r.r/r* layer_idxrr3rrrrr  attn_scale floor_scaleattn_temperature_tuningattention_dropout is_causalno_rope_layersuse_roper5rcattention_biasq_projk_projv_projo_proj use_qk_normrn rms_norm_epsqk_normr?r*r!r@s rAr/Llama4TextAttention.__init__Ds  " F4F4F&JdJd4de #)#=#= $*$>$>&B\B\$\!#)#=#= }}d*  ++!--'-'E'E$!'!9!9--i8 ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii  & & 68J8JQWQfQf   ;; " "t}}+F,?,?@DL(5 "rCNrDposition_embeddingsr past_key_valuesrrEc URSSn/UQSPURP7nURU5RU5nUR U5R"/UQSPURP76n UR U5RU5R SS5n UR(a'[XURUR55up[US5(a"URU5nURU 5n UR(GaUR(dUbURUR5OSn [ R""URSURS9U -n [ R$"[ R&"U R)5S-UR*- 55UR,-S-n U RSUSSS45R//UQSPSP75n X-RUR05nUR SS5nU R SS5n UbUR3XUR5up[4R6"UR8R:[<5nU"UUU U U4UR>(dSO UR@URBS .UD6unnURD"/UQSP76RG5nURIU5nUU4$) NrGr$r-r0rrr)r r )%rKrr*rJr+r,rr(rrrhasattrr0r$get_seq_lengthr!r7rlog1pfloorrr#r"rrupdater get_interfacer*_attn_implementationrrr%r rrr-)r?rDr3r r4r input_shape hidden_shape query_statesrrpast_seen_tokens positions attn_scalesattention_interfacerrs rArRLlama4TextAttention.forwardbs$))#2. 88b8$--8 {{=166|D [[/44UkU2Ut}}U {{=166|DNNqRST =='7*=*@*@ATAT*U( $L 4 # #<< 5Lj1J  ' ' ' Q`Ql==dnnMrs  ]%8%8%;MDXDXY\llI EKK):S)@DDTDT(TUVY]YhYhhknn &**A{21+EFMMNbP[Nb]^Nb`aNbcK(6::<;M;MNL#--a3 ))!Q/  &'6'='=jX\XfXf'g $J(?(M(M KK , ,.E) %8      % $}}C$2H2HLL %  % ! \"));;;;FFH kk+. L((rC)r%r"r$r*r#rr&r+r!rrrr-r*r0r r(r,ri)rTrUrVrW__doc__r&r/r7rXrr rrrRrYrZr[s@rArrAsGA/AF)- 8)||8)#5<<#=>8) t+ 8)  8) -. 8) u||U\\D0% 2E2LL M8)8)rCrcH^\rSrSrU4SjrSS\R S\R S-S\RS-S\S-S\ S-S \ \R \R 4S-S \ \ S \ \R\ \R\R4S-44S jjrS rU=r$)Llama4TextDecoderLayeric>[TU]5 URUlX l[ X5UlX!R ;UlUR(a[U5Ul O[XRS9Ul [URURS9Ul[URURS9Ulg)N)r2r)r.r/r3r!r self_attn moe_layers is_moe_layerr feed_forwardr]intermediate_size_mlprr/input_layernormpost_attention_layernormr1s rAr/Llama4TextDecoderLayer.__init__s !--",V?%):)::    -f 5D  -fHdHd eD 01C1CI\I\](9&:L:LRXReRe(f%rCNrDr rr4 use_cacher3rrEc UnURU5nUR"SUUUUUS.UD6upX-nUnURU5nURU5nUR(aUupXR UR 5-nU$)N)rDr3r r4rS)rPrKrQrNrMrJrK) r?rDr rr4rSr3rresidualattention_states_s rArRLlama4TextDecoderLayer.forwards!,,]; #nn ' 3)+    !3 !55mD ))-8   , M #5#5hnn#EE rC)rNr3rPrMr!rQrK)NNNFN)rTrUrVrWr/r7rX LongTensorr boolrrr FloatTensorrRrYrZr[s@rArIrIs g"/304(,!&HL ||  t+ &&-    $; #5<<#=>E -.  u  %(9(95;L;L(L"MPT"TT U  rCrIc~^\rSrSr%\\S'SrSrS/rSr Sr Sr Sr Sr \R"5U4Sj5rSrU=r$) Llama4PreTrainedModelir*)imagetextTr4Fc>[TU]U5 [URS5(aURROURR Rn[ U[5(aA[R"URSUS9 [R"URSUS9 g[ U[5(a;[R"URURUR55 g[ U[ 5(aS[R"UR"UR$S9 [R"UR&UR$S9 gg)Ninitializer_ranger7)rystd)rc)r. _init_weightsr8r*rb text_configrr(initnormal_r9r;Llama4VisionRotaryEmbeddingcopy_freqs_ci_compute_freqs_ciLlama4VisionModelclass_embeddingscalepositional_embedding_vlm)r?rrcr@s rArd#Llama4PreTrainedModel._init_weightss f%t{{$788 KK ) )((:: f/ 0 0 LL,,3C @ LL)) =  ; < < JJv(@(@(O P  1 2 2 LL//V\\ B LL88fll K3rCrU)rTrUrVrWr%rinput_modalitiessupports_gradient_checkpointing_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_supports_attention_backendr7rrdrYrZr[s@rAr^r^sU (&*##4"5 N!"& ]]_LLrCr^c8^\rSrSr%S/rSrSr\\S'\ \ \ S.r S\4U4Sjjr \\\\SS \R&S-S \R(S-S \R&S-S \S-S \R,S-S\S-S\\S\\-4Sjj5555rSrU=r$)Llama4TextModelirImodel)r`r*) attentionsrDrc >[TU]U5 URUlURUl[ R "URURUR5Ul[ R"[UR5Vs/sHn[X5PM sn5Ul [URURS9Ul[#US9UlSUlUR)5 gs snf)Nrr*F)r.r/ pad_token_id padding_idx vocab_sizer5 Embeddingr3 embed_tokens ModuleListrangenum_hidden_layersrIlayersrr/normr rotary_embgradient_checkpointing post_initr1s rAr/Llama4TextModel.__init__s  !.. ++LL):):FQ>QR 36B&+#  isC?N input_idsr rr4 inputs_embedsrSrrEc LUSLUSL- (a [S5eUc>URURURRR55nU(aUc[ UR S9nUcUUbUR5OSn[R"URSURS9U-nURS5n[U=n [5(d)UR UUUUS.n [S 0U D6[S 0U D6S.n Un UR!X5n [#UR$SUR R&5H-upU"U 4XR R(U UUUU S.UD6n M/ UR+U 5n [-U U(aUS 9$SS 9$) N:You must specify exactly one of input_ids or inputs_embedsr~rr$r6)r*rr r4r)full_attentionchunked_attention)r rr4rSr3)last_hidden_stater4rU) ValueErrorrrrrr r*r9r7rrK unsqueezerdictrrr enumeraterr layer_typesrr)r?rr rr4rrSrrBcausal_mask_mapping mask_kwargsrDfreq_cisi decoder_layers rArRLlama4TextModel.forward s  -t"; <YZ Z   --ill4;L;L;S;S;Z;Z.[\M 0*$++>O  CRC^==?de  << (;(;A(>}G[G[\_ooL'11!4L?-FF++!."0#2 , K#5"C{"C%?%N+%N#  & ??=? )$++6U 8U8U*V W A)2;;3J3J13MN) /#$, M!X -0 &+/8O  >B  rC)rrrrrrr)NNNNNN)rTrUrVrW_no_split_modulesbase_model_prefixrqr&rrrIr_can_record_outputsr/rr"r#rr7rZrXr r\r[rrrrrRrYrZr[s@rArzrzs12  )/& / .2.204(,26!%< ##d*<  t+< &&- <  < ((4/ < $;< +,<  ( (<  < rCrzcd^\rSrSr%S/rSrSS0rSS0r\\ S'S\4U4S jjr \ \ SS \ RS -S \ RS -S \ RS -S\S -S\ R"S -S\ RS -S\S -S\\ R-S\\S\\-4Sjj55rSrU=r$)Llama4ForCausalLMiMrIlanguage_modelzlm_head.weightzmodel.embed_tokens.weightlm_headcolwise_gather_outputr*c>[TU]U5 [U5UlURUl[ R "URURSS9UlUR5 gr`) r.r/rzr{rr5rcr3rrr>s rAr/Llama4ForCausalLM.__init__TsU  $V,  ++yy!3!3V5F5FUS  rCNrr rr4rlabelsrSlogits_to_keeprrEc nUR"SUUUUUUS.U D6n U Sn [U[5(a [U*S5OUn UR U SS2U SS245n SnUb)UR "SXUR RS.U D6n[UU U RU RU RS9$)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, Llama4ForCausalLM >>> model = Llama4ForCausalLM.from_pretrained("meta-llama4/Llama4-2-7b-hf") >>> tokenizer = AutoTokenizer.from_pretrained("meta-llama4/Llama4-2-7b-hf") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```)rr rr4rrSrN)logitsrr)lossrr4rDr|rU) r{rrslicer loss_functionr*rrr4rDr|)r?rr rr4rrrSrroutputsrD slice_indicesrrs rArRLlama4ForCausalLM.forward]sH** )%+'     8B>SV8W8W~ot4]k mA}a,?@A  %%pVt{{OeOepiopD%#33!//))   rC)rr{r)NNNNNNNr)rTrUrVrWrr_tied_weights_keys_tp_planr&rr/rrr7rZrXr r\r[rrrrrrRrYrZr[s@rArrMs&12(*,GH23H /.2.204(,26*.!%-.: ##d*:  t+: &&- :  : ((4/ :   4': $;: ell*: +,:  ' ': : rCrzQ Base class for Llava causal language model (or autoregressive) outputs.  custom_introc\rSrSr%SrSr\RS-\S'Sr \RS-\S'Sr \ S-\S'Sr \ \RS-\S'Sr\ \RS-\S'Sr\RS-\S 'S rg) Llama4CausalLMOutputWithPastia loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size (batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nrrr4rDr|image_hidden_statesrU)rTrUrVrWrGrr7r\rrr4r rDrr|rrYrUrCrArrs &*D%  d ")'+FE   $+$(OUT\(59M5**+d2926Je''(4/648**T18rCrc.^\rSrSrU4SjrSrSrU=r$)Llama4VisionMLP2icx>[TU]5 URUlURUl[R "URUR SS9Ul[R "URURSS9Ul [R"5Ul URUl gr`)r.r/r3r2r5rcprojector_input_dimfc1projector_output_dimfc2GELUrfprojector_dropoutr r>s rAr/Llama4VisionMLP2.__init__s !--!'!9!999T33V5O5OV[\99V88&:U:U\abWWY// rCcURU5nURU5n[R"XRURS9nURUR U55$)Nr )rrfFr rrr?rDs rArRLlama4VisionMLP2.forwardsR/ **=9  -<<$--X !!$((="9::rC)rfr rrr3r2rlr[s@rArrs0;;rCrc.^\rSrSrU4SjrSrSrU=r$)Llama4MultiModalProjectoric>[TU]5 [R"URR UR RSS9Ulgr`) r.r/r5rc vision_configvision_output_dimrer3linear_1r>s rAr/"Llama4MultiModalProjector.__init__s?   2 2    * *  rCc(URU5nU$rir)r?image_featuresrDs rArR!Llama4MultiModalProjector.forwards n5 rCrrlr[s@rArrs rCrc 8URup#n[[R"U55nUR X%US5nUR 5up&ptUR X&[Xq-5[XA- 55nUR SSSS5R5nUR U[Xa-5[Xq-5[XAS-- 55nUR SSSS5R5nUR USURS5n U $)NrGrr-r$r)rKrmathsqrtrJsizepermuter) input_tensor shuffle_ratio batch_size num_patcheschannels patch_sizeheightwidthreshaped_tensor output_tensors rA pixel_shufflers(4(:(:%JXTYY{+,J$$ZZLL*6*;*;*='J"'' C@U[TU]5 URUl[URURS--5UlUR Ul[U5Ul gr,) r.r/pixel_shuffle_ratiorr inner_dimr output_dimrmlpr>s rAr/$Llama4VisionPixelShuffleMLP.__init__sX #)#=#= V77D[R"UR5R"/URSSQSPSP765n[R"UR5R"/URSSQSPSP765n[ X#S9nUR UR5n[R"X2-5RS5n[R"XB-5RS5nURU5URU54$)NrGr-)rjrr) r7rrrrKrrrrrr)rrrjquery_key_ query_outkey_outs rAvision_apply_rotary_embrs  " "5;;=#8#8#R%++cr:J#RB#RPQ#R SF  !4!4!Lciin!Lb!L!!L MD$hEH{{6==)H""6#45==a@I  199![TU]5 XlURUlUR UlURUR -UlSUlURUl URS-Ul [R"URUR UR-SS9Ul [R"URUR UR-SS9Ul[R"URUR UR-SS9Ul[R"UR UR-URSS9Ulg)Nr$rTra)r.r/r*r3 embed_dimr num_headsrrr%r r5rcr*r+r,r-r>s rAr/Llama4VisionAttention.__init__s  ++33**f.H.HH $%!!'!9!9}}d* ii0NUYZ ii0NUYZ ii0NUYZ ii >UYZ rCNrDrjr r4rrEc URSSn/UQSPURP7nURU5RU5nUR U5RU5n UR U5RU5n [ XUS9upURSS5nU RSS5n U RSS5n [R"URR[5n U "UUU U S4UR(dSO URSSS.UD6upU R"/UQSP76R!5n UR#U 5n X4$)NrG)rjr$r-r7F)r r r&)rKrr*rJr+r,rrrr=r*r>rrr%rrr-)r?rDrjr r4rr?r@rArrrErrs rArRLlama4VisionAttention.forwards`$))#2. 88b8$--8 {{=166|D [[/44\B {{=166|D #:<^f#g #--a3 ))!Q/ #--a3 (?(M(M KK , ,.L) %8      % $}}C$2H2H %  % ! "));;;;FFH kk+. ((rC) r%r*rrr+rrr-r*r r,NN)rTrUrVrWrr/r7rXr rrrrRrYrZr[s@rArrs [1 [&/3(, ')||'),,') t+ ')  ') -. ') u||U\\D0% 2E2LL M')')rCrcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)Llama4VisionMLPiIc>[TU]5 Xl[R"5Ul[R "URURSS9Ul [R "URURSS9Ul g)NTra) r.r/r*r5rrfrcr3r2rrr>s rAr/Llama4VisionMLP.__init__Jsc  WWY99V//1I1IPTU99V55v7I7IPTUrCrDrEclURU5nURU5nURU5nU$ri)rrfrrs rArRLlama4VisionMLP.forwardQs4/ **=9 / rC)rfr*rrrr[s@rArrIs)VU\\ellrCrc ^\rSrSrS\4U4SjjrS S\RS\RS\RS-S\S-4S jjr S r U=r $) Llama4VisionEncoderLayeriXr*c>[TU]5 URUl[U5Ul[ U5Ul[R"UR5Ul [R"UR5Ul gri) r.r/r3rrKrrr5 LayerNormrPrQr>s rAr/!Llama4VisionEncoderLayer.__init__Ysb !--.v6"6*!||F,>,>?(* V5G5G(H%rCN hidden_staterjr output_attentionscUnURU5nURUUUS9upXQ-nUnURU5nURU5nXQ-nU4nU(aXv4- nU$)N)rjr )rPrKrQr)r?r rjr rrVrrs rArR Llama4VisionEncoderLayer.forwardcs ++L9 %)^^ )&4& "  .  44\B xx - . /   &GrC)r3rPrrQrKr) rTrUrVrWrr/r7rXr[rRrYrZr[s@rAr r Xs_I1I/3)- ll,, t+   $; rCr c^\rSrSrSrS\4U4SjjrSS\RS\RS\RS-S \ S-S \ S-S \ S-S \ \ -4S jjr Sr U=r$)Llama4VisionEncoderiz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`Llama4VisionEncoderLayer`]. Args: config: Llama4VisionConfig r*c>[TU]5 Xl[R"[ UR 5Vs/sHn[U5PM sn5UlSUl Xlgs snf)NF) r.r/r*r5rrrr rr)r?r*rXr@s rAr/Llama4VisionEncoder.__init__sY  mmuU[UmUmOn$oOn!%=f%EOn$op &+# %psA,NrDrjr routput_hidden_states return_dictrEcUbUOURRnUbUOURRnUbUOURRnU(aSOSnU(aSOSnURH,n U(aXq4-nU "UUUUS9n U(aXS4-nU SnM. U(aXq4-nU(d[ SXU455$[ XUS9$)a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NrU)r r rrjr$rc3.# UH ocMUv M g7frirU.0vs rA .Llama4VisionEncoder.forward..se$Sq$S rrDr|)r*rrrrrr) r?rDrjr rrrencoder_statesall_attentions encoder_layer layer_outputss rArRLlama4VisionEncoder.forwards>2C1N-TXT_T_TqTq$8$D $++JjJj &1%>Ne]N$See e+Vd  rC)r*rrNNNN)rTrUrVrWrGrr/r7rXr[rrrRrYrZr[s@rArrs1/3)-,0#'? ||? ,,?  t+ ?  $; ? #Tk ? D[?   ? ? rCrcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)Llama4UnfoldConvolutionic8>[TU]5 URn[U[5(aX"4n[ R RX!RS9Ul[ R"URUS-US-URSS9Ul g)N) kernel_sizestriderr$Fra) r.r/rrrr7r5Unfoldunfoldrc num_channelsr3linear)r?r*r)r@s rAr/ Llama4UnfoldConvolution.__init__s '' k3 ' '&4Khhoo+FWFWoX ii   +a. 0;q> A     rCrDrEcpURU5nURSSS5nURU5nU$)Nrr-r$)r,rr.rs rArRLlama4UnfoldConvolution.forwards8 M2 %--aA6  M2 rC)r.r,rr[s@rAr'r's(  U\\ellrCr'cF^\rSrSrS\4U4Sjjr\S5rSrSr U=r $)rhir*cn>[TU]5 XlURSUR U5SS9 g)NrjFr)r.r/r*rrkr>s rAr/$Llama4VisionRotaryEmbedding.__init__s4   Z)?)?)GTYZrCcURUR-n[R"US-[RS9R US-S5n[R "X"SS/SS9nSUS'X!-nX!-nURUR-S-nSURS [R"SUS5SUS-R5U- -- nUS-S USSSS24-RSS S9nUS-S USSSS24-RSS S9n[R "Xx/S S9R5R5S SSS24n U RUR S SS5S:S5n [R"[R"[R "U 5[R""U 5/S S95n U $) Nr-rr$rrH)rGrGrr).NrG.) image_sizerr7rint32rcatr3rrrrepeat_interleaver masked_fillrstackcossin) r*idximg_idx frequencies_x frequencies_yfreq_dim rope_freqfreqs_xfreqs_yrrs rArk-Llama4VisionRotaryEmbedding._compute_freqs_cis6#4#44,,sAvU[[9AA#q&!L))Wbqk2:  %%)C)CCqH  " "< 0 Q!,->A ?EEG(R T "A%y1IdD!m4LL__`agi_j!A%y1IdD!m4LL__`agi_j 7,"5;;=HHJ3PSRSPS8T!!'//"a";a"?C((eii6F RWHX5Y_a)bcrCcLURRUR5$ri)rjrrrs rArR#Llama4VisionRotaryEmbedding.forwards}} 4 455rCr~) rTrUrVrWrr/rrkrRrYrZr[s@rArhrhs0[1[ &66rCrhc^\rSrSr%SrSrS/r\\S'S\4U4Sjjr Sr SS \ RS \ RS-S \ S-S \ S-S \ S-S\\\ RS4-4 SjjrSrU=r$)rli vision_model)r_r r*c~>[TU]U5 URUlURUlURUlUR UlURUR-S-S-UlURS-Ul[U5Ul [R"UR[R"UR5-5Ul[R"UR[R"UR UR5-5Ul[!U5Ul[R$"UR5Ul[R$"UR5Ul[+U5Ul[/U5UlUR35 g)Nr-r$r)r.r/r7rr3r-rrnr'patch_embeddingr5r6r7randnrmrorhrotary_embeddingr  layernorm_prelayernorm_postrr{rvision_adapterrr>s rAr/Llama4VisionModel.__init__sA   ++ ++!--"// OOt>1DqH''- 6v>!||DJJTEUEU9V,VW(* TZZ%++dN^N^`d`p`pBq5q(r% ;F C \\$*:*:; ll4+;+;<)0 9&A rCcUR$)zW This function is used to fetch the first embedding layer to activate grads on inputs. )rMrs rAget_input_embeddings&Llama4VisionModel.get_input_embeddings's###rCN pixel_valuesr rrrrE.c UbUOURRnUbUOURRnUbUOURRnURupxpSn Sn UR U5n U RupnU R X{-U -UU5n URRU RSSU RS5n[R"U U/SS9n US- nU R X{-XU5n URRU RU RS9nU U-n URU 5n U R!USU5n UR#U5nUR%U SUUUS9nUR&n UR)U 5n U SS2SS2SS24n UR+U 5n U(a UR,OSnU(aUSnOSnU(d[/S U UU455$[1U UUS 9$) a Example: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, MllamaVisionModel >>> checkpoint = "meta-llama/Llama-3.2-11B-Vision" >>> model = MllamaVisionModel.from_pretrained(checkpoint) >>> processor = AutoProcessor.from_pretrained(checkpoint) >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, return_tensors="pt") >>> output = model(**inputs) >>> print(output.last_hidden_state.shape) torch.Size([1, 1, 4, 1025, 7680]) ``` Nr$rrGrHrr)r rrrjr-c3.# UH ocMUv M g7frirUrs rAr,Llama4VisionModel.forward..s_$Mq$Mrr)r*rrrrKrMrrmrr7r9rorrrrPrJrOr{rrQrRrDrr)r?rWr rrrrbatch_size_times_num_tilesr-rrnum_concurrent_media num_chunksr rXrrrmpositional_embeddingrjrrDr|s rArRLlama4VisionModel.forward-sFD2C1N-TXT_T_TqTq$8$D $++JjJj &1% >b b rCrlc#^\rSrSr%SS/r0rSr\\S'S\4U4Sjjr Sr Sr S r S r S rS r\\"S S9\"SS9S\R(S\S\\S\\-4Sj555rS\R6S\R(S\R(4Sjr\\"S S9\S'S\R6S-S\R(S-S\R:S-S\R6S-S\S-S\R(S-S\S-S\R6S-S\S-S \S-S!\S-S"\S-S#\ \R:-S\\S\\!-4S$jj555r"S(S%jr#S&r$U=r%$))Llama4ForConditionalGenerationirIr r{r*c>[TU]U5 [UR5Ul[ U5Ul[UR5Ul URRUl [URS5(aURRUl O.URRR=(d SUl UR5 g)NrrG)r.r/rlrrKrmulti_modal_projectorrrerrr8r*rrr>s rAr/'Llama4ForConditionalGeneration.__init__s  -f.B.BC%>v%F"/0B0BC ,,77 4;; / / $ 8 8D  $ 7 7 D D JD  rCc6URR5$ri)rrUrs rArU3Llama4ForConditionalGeneration.get_input_embeddingss""7799rCc:URRU5 gri)rset_input_embeddings)r?rs rAri3Llama4ForConditionalGeneration.set_input_embeddingss 007rCc6URR5$ri)rget_output_embeddingsrs rArl4Llama4ForConditionalGeneration.get_output_embeddingss""88::rCc:URRU5 gri)rset_output_embeddings)r?new_embeddingss rAro4Llama4ForConditionalGeneration.set_output_embeddingss 11.ArCc:URRU5 gri)r set_decoder)r?decoders rArs*Llama4ForConditionalGeneration.set_decoders ''0rCc6URR5$ri)r get_decoderrs rArw*Llama4ForConditionalGeneration.get_decoders""..00rCF)tie_last_hidden_stateszOObtains image last hidden states from the vision tower and apply al projection.rrWvision_feature_select_strategyrrEc UR5VVs0sH upEUcM XE_M nnnUR"U40UD6$s snnf)a: pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`) The tensors corresponding to the input images. vision_feature_select_strategy (`str`): The feature selection strategy used to select the vision feature from the vision backbone. Can be one of `"default"` or `"full"` )itemsrK)r?rWrzrkrs rAget_image_features1Llama4ForConditionalGeneration.get_image_featuressB $*<<>C>41Q$!$>C  888Ds ==rrrc "UcjX R5"[R"URR[R UR S95:HnURS5nOXRR:HnUR5nURS5RU5RUR 5n[X$R5UR5:HSUSURS35 U$)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. rYrGz6Image features and image tokens do not match, tokens: z , features: r)rUr7tensorr*image_token_idlongrallrr expand_asrr numelrK)r?rrrspecial_image_maskn_image_tokenss rAget_placeholder_mask3Llama4ForConditionalGeneration.get_placeholder_masks  !.2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !*kk.H.H!H +//1/99"=GG VYYZgZnZno  - 3 3 59M9M9O OD^DTT`aoauauvwax`y z "!rCNr rr4rrSrrrrc U bU OURRn U bU OURRn U bU OURRn USLUSL- (a [ S5eUbUb [ S5eUcUR 5"U5nUbUR UUSS9RnURSURS55nURU5RURUR5nURXUS9nURUU5nUR "S UUUUU U U U U S. UD6nUS nSnUGb>UbUSS2UR"S S - *S24RUR5nUS SS2SS24URUR5S :gR%5nUS S S24URUR5S :gR%5nO1US SS2SS24R%5nUS S S24R%5n[&R("5nU"URSURS55URS5RUR55nU (dU4US S-nUbU4U-$U$[+UUUR,UR.UR0UbWS 9$SS 9$)aA labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, LlavaForConditionalGeneration >>> model = LlavaForConditionalGeneration.from_pretrained("llava-hf/llava-1.5-7b-hf") >>> processor = AutoProcessor.from_pretrained("llava-hf/llava-1.5-7b-hf") >>> prompt = "USER: \nWhat's the content of the image? ASSISTANT:" >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, text=prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(**inputs, max_new_tokens=15) >>> processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "USER: \nWhat's the content of the image? ASSISTANT: The image features a busy city street with a stop sign prominently displayed" ```NrzdYou cannot specify both pixel_values and inputs_embeds at the same time, and must specify either oneT)rWrzrrG)rr) r rr4rrSrrrrrr$.)rrr4rDr|rrU)r*rrrrrUr~rrJrrdrrrrmasked_scatterrrKrr5CrossEntropyLossrr4rDr|)r?rrWr rr4rrzrrSrrrrrr vision_flatprojected_vision_flatrrrrshift_attention_mask shift_logits shift_labelsloss_fctrs rArR&Llama4ForConditionalGeneration.forwardsKd2C1N-TXT_T_TqTq$8$D $++JjJj &1%$>{$K$N$N$$m&9&9% !"&!:!:G\";" *889KMbcM%%  )%+'/!5#)      )(6a6<<?Q;N9O9Q6Q'R'U'UV\VcVc'd$%c3B3k23G3J3J6==3Y]^3^_jjl %c12g/C/F/Fv}}/UYZ/Z[ffh %c3B3k2==? %c12g99; **,H!!"l&7&7&;O>OPR>S>V>VWcWjWj>kDY,F'+'7D7V# CV C+#33!//))2>2J   QU   rCc URR"U4UUUUUS.UD6n U(dURSS5(dXIS'U $)N)r4rr ris_first_iterationrSTrW)rprepare_inputs_for_generationget) r?rr4rrWr rrr model_inputss rArrs $! N&!.)7KB9G&jjG5@   H % B)BII)$ !uxx != =( ,299 ,_-"BII".",? ?D 2  2  2|| 2 5<< %& 2 UU\\ U# U%,, U(% II% <<% % << % LL4' %  %%B% II% <<% % << % LL4' %  %%2Y)"))Y)x/7/dLOLL<[ +[ [ |L -L ^  9;9  90;uxx;"  ( )")) )!ELL!! : << :  :ll : 5<< %& :7)BII7)t bii )9)XO "))O dbii(6"))6