Zja^SrSSKJr SSKJr SSKrSSKJr SSKJr SSK J r SS K J r SS K Jr SS KJr SS KJrJr SS KJr SSKJr SSKJrJrJrJrJrJr SSKJ r SSK!J"r" SSK#J$r$ \RJ"\&5r'\\"SS9"SS\555r(\"SS9\"SS\555r)"SS\RT5r+S\RXS\4S jr-\ "S!S"S#S$9S6S%\ S#\RXS&\RXS-S'\S-S(\RXS-S)\RXS-S*\R\S-S+\/S-S,\/S-S\04S-jj5r1\"S.S/\55r2\"S0S9"S1S2\255r3\"S0S9"S3S4\2\ 55r4/S5Qr5g)7zPyTorch PaliGemmamodel.)Callable) dataclassN)nn)Cache)PreTrainedConfig)GenerationMixin)create_masks_for_generate)FlashAttentionKwargs)BaseModelOutputWithPastBaseModelOutputWithPooling)PreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tupleloggingtorch_compilable_check)deprecate_kwarg) AutoModel)PaliGemmaConfigzN Base class for Paligemma outputs, with hidden states and attentions.  custom_introcB\rSrSr%SrSr\RS-\S'Sr g)PaligemmaModelOutputWithPast.a image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states) __name__ __module__ __qualname____firstlineno____doc__r torch FloatTensor__annotations____static_attributes__r!ځ/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/paligemma/modeling_paligemma.pyrr.s 59**T18r+rzU Base class for PaliGemma causal language model (or autoregressive) outputs. c\rSrSr%SrSr\RS-\S'Sr \RS-\S'Sr \ S-\S'Sr \ \RS-\S'Sr\ \RS-\S'Sr\RS-\S 'S rg) PaliGemmaCausalLMOutputWithPast>a loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.text_config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder after projecting last hidden state. Nlosslogitspast_key_values hidden_states attentionsr r!)r"r#r$r%r&r0r'r(r)r1r2rr3tupler4r r*r!r+r,r.r.>s &*D%  d ")'+FE   $+$(OUT\(59M5**+d2926Je''(4/648**T18r+r.c6^\rSrSrS\4U4SjjrSrSrU=r$)PaliGemmaMultiModalProjector\configc>[TU]5 [R"URR URR SS9Ulg)NTbias)super__init__rLinear vision_config hidden_sizeprojection_dimlinearselfr9 __class__s r,r>%PaliGemmaMultiModalProjector.__init__]s; ii 4 4 @ @&BVBVBeBelpq r+c(URU5nU$NrC)rEimage_featuresr3s r,forward$PaliGemmaMultiModalProjector.forwardas N3 r+rJ) r"r#r$r%rr>rLr* __classcell__rFs@r,r7r7\srrr+r7 group_idsreturnc T^S[S[S[S[S[4 U4SjjnU$)aY This function adds the correct offsets to the `q_idx` and `kv_idx` as the torch API can only accept lengths, not start and end indices. Args: group_ids (`torch.Tensor`): A tensor of shape `(bs, len)` assigning each token to a vision group. Tokens with the same group come from the same input image. Text is denoted by `-1`. batch_idxhead_idxq_idxkv_idxrQc>T RSnURUS- S9nURUS- S9nT X4nT X4n[R"X$:US5n[R"X4:US5nXx:HUS:-$)Nr)maxr)shapeclampr'where) rSrTrUrV seq_length q_idx_clampedkv_idx_clampedq_groupkv_grouprPs r, inner_mask0token_type_ids_mask_function..inner_maskqs__R(  Q 7 *q.9I45Y67++e0'2>;;v2HbA#1 55r+)intbool)rPrbs` r,token_type_ids_mask_functionrfgs3 6c 6S 6 6c 6d 6 r+ input_embeds5.6.0 inputs_embedsversionnew_namer9attention_maskr2 position_idstoken_type_ids pixel_values is_trainingis_first_iterationc  U(aUc [S5eUR5UUUUS.n U(aUO&USL=(d UR(+=(d USLnU(dU RSS5(d>UbSU- nO5[R S5 [ R"U5SS2SS2S4nUbU(aUS:HRUR5n [RRU S SS 9SS2SS 24n X)-n [ R"U R5SS 9S- n[ R"X[ R "US 55n[#U5U S '[%S0U D6$)a Overwrites the base `create_masks_for_generate` with `token_type_ids` masking to create the causal mask mapping for all kinds of forward passes. Paligemma uses a bidirectional mask on the prompt tokens. Uses `pixel_values` as an optional input to disambiguate edge cases. Nz;`token_type_ids` is required as a model input when training)r9rirmr2rn use_cacheTrzIt is a prefill stage but The `token_type_ids` is not provided. We recommend passing `token_type_ids` to the model to prevent bad attention masking.r)rr)valuerX)dimor_mask_functionr!) ValueErrorget_text_configis_initializedgetlogger warning_oncer' ones_liketodevicer functionalpadcumsumrdr\ full_likerfr )r9rirmr2rnrorprqrrkwargs mask_kwargsis_imageis_previous_imagenew_image_startrPs r,create_causal_mask_mappingrss&~-VWW((*&(*$ K  %g_-K-K)Kg|cgOg K!>!>  %/N   Z  #__];Aq!GDN !&8 #a'++M,@,@AMM--ha-HCRCP"%77LL!4!4!6A>B KKU__^UW5XY *Fy*Q &' $ 3{ 33r+cJ\rSrSr%\\S'SrSrSrS/r Sr Sr Sr Sr SrSrS rg ) PaliGemmaPreTrainedModelr9model)imagetextTr7r2Fr!N)r"r#r$r%rr)base_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_can_compile_fullgraph_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendr*r!r+r,rrsF (&*#78"3"N"&r+rz| The Base Paligemma model which consists of a vision backbone and a language model without language modeling head., c^\rSrSrSrS\4U4SjjrSrSr\ \ "SS9S \ RS \ \S \\-4S j55rS \ R$S\ RS\ R4Sjr\ \ SS \ R$S-S \ RS-S\ R(S-S\ R$S-S\S-S\ R$S-S\ RS-S\ R$S-S\S-S \ \S \\-4Sjj55rSrU=r$)PaliGemmaModelFr9c>[TU]U5 [R"URS9Ul[ U5UlURRUl [R"URS9nX l URR5R=(d URUlUR5 g)N)r9)r=r>r from_configr@ vision_towerr7multi_modal_projector text_config vocab_sizelanguage_modelr9rydtypetext_config_dtype post_init)rEr9rrFs r,r>PaliGemmaModel.__init__s  %119M9MN%A&%I" ,,77"..f6H6HI,!%!!D!D!R  r+c6URR5$rI)rget_input_embeddings)rEs r,r#PaliGemmaModel.get_input_embeddingss""7799r+c:URRU5 grI)rset_input_embeddings)rErus r,r#PaliGemmaModel.set_input_embeddingss 007r+zWObtains image last hidden states from the vision tower and apply multimodal projection.rrprrQc rUR"U40UD6nURnURU5nXSlU$rI)rlast_hidden_stater pooler_output)rErpr image_outputsselected_image_featurerKs r,get_image_features!PaliGemmaModel.get_image_featuressB)),A&A !.!@!@334JK&4#r+ input_idsrirKc FUcjX R5"[R"URR[R UR S95:HnURS5nOXRR:HnUR5nURSURS-nURS5RU5RUR 5n[X$R5UR5:HSUSU35 U$)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. )rrrXrrz6Image features and image tokens do not match, tokens: z , features: )rr'tensorr9image_token_idlongrallsumrZ unsqueeze expand_asrrnumel)rErrirKspecial_image_maskn_image_tokensn_image_featuress r,get_placeholder_mask#PaliGemmaModel.get_placeholder_masks   !.2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !*kk.H.H!H +//1)//2^5I5I!5LL/99"=GG VYYZgZnZno  - 3 3 59M9M9O OD^DTT`aq`r s "!r+Nrmrnr2rolabelsrtc USLUSL- (a [S5eUbQURRUR:a-XRR:Hn UR 5n SX'OUn UcUR 5"U 5nUcXUbUR 5OSn [R"URSURS9U -nURS5S-nUbbURU5RnURURUR5nUR!XUS9n UR#X5n[%U=n[&5(d$[)URUUUUUUUR*S9nUR,"S UUUUU S.U D6n[/UR0UR2UR4UR6UbWS 9$SS 9$)  labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.text_config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.text_config.vocab_size]`. Example: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, PaliGemmaForConditionalGeneration >>> model = PaliGemmaForConditionalGeneration.from_pretrained("google/paligemma2-3b-mix-224") >>> processor = AutoProcessor.from_pretrained("google/paligemma2-3b-mix-224") >>> prompt = "Where is the cat standing?" >>> url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, text=prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(**inputs,) >>> processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Where is the cat standing?\nsnow" ```Nz:You must specify exactly one of input_ids or inputs_embedsrr)r)rirK)rq)rmrnr2rirt)rr2r3r4r r!)rxr9rrclonerget_seq_lengthr'arangerZrrrrrrrmasked_scatter isinstancedictrtrainingrrrr2r3r4)rErrprmrnr2rorirrtrr llm_input_idspast_seen_tokensrKcausal_mask_mappingoutputss r,rLPaliGemmaModel.forwardsZ  -t"; <YZ Z  T[[%?%?4??%R!*kk.H.H!H %OO-M01M -%M   557 FM  CRC^==?de  << (;(;A(>}G[G[\_ooL'11!4q8L  #!44\BPPN+..}/C/C]EXEXYN!%!:!:~";" *889K\M?-FF"<  MM # %% .%+'    ,%77#33!//))2>2J   QU   r+)rrrrr) NNNNNNNNN)r"r#r$r%accepts_loss_kwargsrr>rrrrr'r(rrr5r r LongTensorrTensorrrer rrLr*rNrOs@r,rrs   :8n!--9?@R9S + +"))":?:K:K"]b]n]n"0.215.204(,2626*.!%c ##d*c ''$.c  t+ c &&- c  c ((4/c ((4/c   4'c $;c -.c  - -c c r+rc^\rSrSrSS0rS\4U4Sjjr\S\RS\ \ 4Sj5r \ \SS \RS -S\RS -S \RS -S \RS -S \S -S\RS -S\RS -S\RS -S\S -S\\R-S\ \ S\\-4Sjj55rSU4Sjjr\\"SSSS9SS\S\RS \RS -S \S -S \RS -S\RS -S\S -S\4Sjj55rSrU=r$)!PaliGemmaForConditionalGenerationizlm_head.weightz(model.language_model.embed_tokens.weightr9c>[TU]U5 [U5Ul[R "UR RUR RSS9Ul UR5 g)NFr;) r=r>rrrr?rrArlm_headrrDs r,r>*PaliGemmaForConditionalGeneration.__init__sS  #F+ yy!3!3!?!?ASASA^A^ejk  r+rprc <URR"U40UD6$rI)rr)rErprs r,r4PaliGemmaForConditionalGeneration.get_image_featuresszz,,\DVDDr+Nrrmrnr2rorirrtlogits_to_keeprQc UR"SUUUUUUUU US. U D6n U Sn [U [5(a [U *S5OU nUR U SS2USS245nSnUb3UR "SXUR RRS.U D6n[UUU RU RU RU RS9$)r) rrprormrnr2rirtrrN)r1rr)r0r1r2r3r4r r!)rrrdslicer loss_functionr9rrr.r2r3r4r )rErrprmrnr2rorirrtrrrr3 slice_indicesr1r0s r,rL)PaliGemmaForConditionalGeneration.forwardsZ**  %))%+'      8B>SV8W8W~ot4]k mA}a,?@A  %%9P9P9[9[_eD/#33!//)) ' ; ;   r+c >[TU]"U4UUUUUU UU S.U D6n U RS5b U SS-U S'U (dU(dX]S'U $)N)r2rirmrnrtrrorrrnrrp)r=prepare_inputs_for_generationr{)rErr2rirnrprmrortrrrrr model_inputsrFs r,r?PaliGemmaForConditionalGeneration.prepare_inputs_for_generationsz w<   +')%))1       N + 7+7+G!+KL ( Y+7 (r+rgrhrjrrc [UUUUUU4SU0UR5VV s0sHupUS:wdM X_M sn nD6$s sn nf)Nrrrp)ritems) r9rirmr2rnrorrrkvs r,r ;PaliGemmaForConditionalGeneration.create_masks_for_generates_*         2  !' F!~2EtqtF  Gs ??)rr) NNNNNNNNNr) NNNNNNTNNF)NF)r"r#r$r%_tied_weights_keysrr>rr'r(rrrrrrrrerdr5r.rLr staticmethodrrrr r*rNrOs@r,rrs? +,VW Eu/@/@EFSeLfEE.215.204(,2626*.!%-.J ##d*J ''$.J  t+ J &&- J  J ((4/J ((4/J   4'J $;J ell*J +,J  0 0J J ^ )V^WO/3*/   ||  t+   llT)   t+  !4K   P r+r)rrr)NNFN)6r&collections.abcr dataclassesrr'r cache_utilsrconfiguration_utilsr generationr masking_utilsr modeling_flash_attention_utilsr modeling_outputsr r modeling_utilsrprocessing_utilsrutilsrrrrrrutils.deprecationrautorconfiguration_paligemmar get_loggerr"r|rr.Moduler7rrfr(rerrrrr__all__r!r+r,rs!$!  3)6BS-&14   H %  9#:9  9  9k9  90299ELLX6?K+/-1$&*C4 C4<<C4LL4'C4T\ C4 ,,% C4 LL4' C4##d*C4C4t C4 C4LC4L ' ' ' c -c  c L \ (@/\  \ ~ ^r+