ZjBSSKJr SSKrSSKJr SSKJr SSKJr SSK J r SSK J r Jr SSKJrJr SS KJr SS KJrJrJr S S KJrJrJrJrJrJr S S KJ r S SK!J"r"J#r# \RH"\%5r&\"SS9\"SS\ 555r'"SS\5r("SS\"5r)"SS\5r*"SS\5r+"SS\5r,"SS\5r-\"SS \55r."S!S"\5r//S#Qr0g)$)CallableN)strict)initialization)PreTrainedConfig)BaseModelOutputBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringlogging)CLIPMLP CLIPAttention CLIPEncoderCLIPEncoderLayerCLIPVisionEmbeddingsCLIPVisionModel)eager_attention_forward)VisionRotaryEmbeddingapply_rotary_pos_emb_visionz&DeepGlint-AI/mlcd-vit-bigG-patch14-336) checkpointc\rSrSr%SrSrSrSr\\ S'Sr \\ S'S r \\ S 'S r \\ S 'S r \\ S'Sr\\ S'Sr\\\-\\\4-\ S'Sr\\\-\\\4-\ S'Sr\\ S'Sr\\ S'Sr\\-\ S'Sr\\ S'Sr\\ S'Srg )!MLCDVisionConfig)aF num_key_value_groups (`int`, *optional*, defaults to 1): Number of key-value groups used in Attention. Example: ```python >>> from transformers import MLCDVisionConfig, MLCDVisionModel >>> # Initializing a MLCDVisionConfig with DeepGlint-AI/mlcd-vit-bigG-patch14-336 style configuration >>> configuration = MLCDVisionConfig() >>> # Initializing a MLCDVisionModel (with random weights) from the DeepGlint-AI/mlcd-vit-bigG-patch14-336 style configuration >>> model = MLCDVisionModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```mlcd_vision_model vision_configi hidden_sizei intermediate_size0num_hidden_layersnum_attention_headsnum_key_value_groupsr num_channelsiP image_size patch_sizegelu hidden_actgh㈵>layer_norm_epsattention_dropoutg{Gz?initializer_range?initializer_factorN)__name__ __module__ __qualname____firstlineno____doc__ model_typebase_config_keyr int__annotations__r!r#r%r'r(r)listtupler+r-strr.floatr0r1r3__static_attributes__r4v/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/mlcd/modular_mlcd.pyrr)s &%J%OK!s!s!! !#!L#47Jd3i%S/1746Jd3i%S/16J NE %(us{(#u# ##rCrc\rSrSrSrg)MLCDMLPQr4N)r5r6r7r8rBr4rCrDrFrFQsrCrFc>\rSrSrS\S\S\R 4SjrSrg)MLCDRotaryEmbeddingUnum_patches_heightnum_patches_widthreturnc~[R"XRRS9R S5R SU5n[R"X RRS9R S5R US5n[R "UR5UR5/SS9n[X5n[R"X`RRURRS9n[R"XpR5nXRS5n U $)aE Calculate the Rotary Position Embedding (RoPE) for MLCDVisionModel based on the grid size. Args: num_patches_height (int): Number of patches in the height dimension. num_patches_width (int): Number of patches in the width dimension. Returns: torch.Tensor: Rotary positional embeddings for the given grid size. )devicer&rdim)rOdtype) torcharangeinv_freqrO unsqueezeexpandstackflattenmaxrSouter) selfrKrLhpos_idswpos_idspos_ids max_grid_sizeseqrotary_pos_emb_fullrotary_pos_embs rDforwardMLCDRotaryEmbedding.forwardVs LL+MM4H4H I S STU V ] ]^`bs t  LL*==3G3G H R RST U \ \]oqs t  ++x//183C3C3EFBO.B ll=1E1ET]]M`M`a#kk#}}=-5==a@rCr4N) r5r6r7r8r<rTTensorrerBr4rCrDrIrIUs ##%,,rCrIcj^\rSrSrS\4U4SjjrS\RS\R4Sjr Sr U=r $)MLCDVisionEmbeddingswconfigc(>[TU]U5 U?gN)super__init__position_embeddingr]rk __class__s rDroMLCDVisionEmbeddings.__init__xs   #rC pixel_valuesrMcHURSnURRRnURUR US95nUR S5R SS5nURRUSS5n[R"XT/SS9nU$)NrrSrr&rPrQ) shapepatch_embeddingweightrStorZ transposeclass_embeddingrXrTcat)r]rt batch_size target_dtype patch_embeds class_embeds embeddingss rDreMLCDVisionEmbeddings.forward|s!''* ++2288 ++LOO,O,OP #++A.88A> ++22:q"E YY ;C rCr4) r5r6r7r8rrorT FloatTensorrgrerB __classcell__rrs@rDririws2$/$ E$5$5 %,,  rCric^\rSrSrSrS\4U4SjjrS S\RS\ \R\R4S\RS-S \ \ S \ \R\RS-44 S jjr S r U=r$) MLCDAttentionzMulti-headed attention with RoPE. Refer to papers: - Attention is all you need: https://huggingface.co/papers/1706.03762 - RoFormer: Enhanced Transformer with Rotary Position Embedding: https://huggingface.co/papers/2104.09864 rkcT>[TU]U5 URUlSUlg)NF)rnror' is_causalrqs rDroMLCDAttention.__init__s%  $*$?$?!rCN hidden_statesposition_embeddingsattention_maskkwargsrMc URSSupVURU5RXVURUR45nUR U5RXVURUR45nUR U5RXVURUR45n USRS5R5n USRS5R5n [XxX5upxURSSSS5R5nURSSSS5R5nU RSSSS5R5n [R"URR[ 5n U "UUUU U4UR"(dSO UR$UR&UR(S.UD6upU RSSSS5R5n U R+XeS5n UR-U 5n U RSSS5R5n X4$)NrPrr&rrr/)dropoutscalingr)rwq_projreshape num_headshead_dimk_projv_projrWrArpermute contiguousr get_interfacerk_attn_implementationrtrainingrscalerviewout_proj)r]rrrrr~ seq_length query_states key_states value_statescossinattention_interface attn_output attn_weightss rDreMLCDAttention.forwards&"/!4!4Sb!9 {{=199:SWSaSacgcpcp:qr [[/77QUQ_Q_aeanan8op {{=199:SWSaSacgcpcp:qr "!$..q1779!!$..q1779#>|Y\#b $++Aq!Q7BBD ''1a3>>@ #++Aq!Q7BBD (?(M(M KK , ,.E) %8      % $}}C$,,JJnn %  % ! "))!Q15@@B !&&zrB mmK0 !))!Q2==? ((rC)rr'rm)r5r6r7r8r9rrorTrgr?r r rerBrrs@rDrrs//3 ,)||,)#5<<#=>,) t+ ,) +, ,) u||U\\D00 1 ,),)rCrc ^\rSrSrS\4U4SjjrS S\RS\\R\R4S\RS-S\ \ S \\R4 S jjr S r U=r$) MLCDEncoderLayerrkcD>[TU]U5 [U5Ulgrm)rnror self_attnrqs rDroMLCDEncoderLayer.__init__s  &v.rCNrrrrrMc UnURU5nUR"SUUUS.UD6upXQ-nUnURU5nURU5nXQ-nU$)ah Args: hidden_states (`torch.FloatTensor`): Input to the layer of shape `(batch, seq_len, embed_dim)`. Represents the hidden states from the previous layer or the input embeddings. position_embeddings (`tuple[torch.Tensor, torch.Tensor]`): A tuple of two tensors, each of shape `(batch, seq_len, embed_dim)`. Represents absolute positional embeddings for the query and key in the attention mechanism. attention_mask (`torch.FloatTensor`): Attention mask of shape `(batch, 1, q_len, k_v_seq_len)` where padding elements are indicated by very large negative values. )rrrr4) layer_norm1r layer_norm2mlp)r]rrrrresidual_s rDreMLCDEncoderLayer.forwards}$!((7 >> ' 3)    !0  ((7 /  0 rC)rrm)r5r6r7r8rrorTrgr?r r rrerBrrs@rDrrs////3 "||"#5<<#=>" t+ " +, " u  ! ""rCrc ^\rSrSrSrS\4U4SjjrS S\RS\ \R\R4S\RS-S \ \ S \ \ -4 S jjrS rU=r$) MLCDEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`MLCDEncoderLayer`]. Args: config: MLCDVisionConfig rkc$>[TU]U5 g)z3Overwrite dummy `MLCDConfig` to `MLCDVisionConfig`.N)rnrorqs rDroMLCDEncoder.__init__s  rCN inputs_embedsrrrrMc TUnURHnU"UUU40UD6nM [US9$)a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. position_embeddings (`tuple[torch.Tensor, torch.Tensor]`): A tuple of two tensors, each of shape `(batch, seq_len, embed_dim)`. Represents absolute positional embeddings for the query and key in the attention mechanism. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) )last_hidden_state)layersr)r]rrrrr encoder_layers rDreMLCDEncoder.forwardsF,& ![[M)# M)+  rCr4rm)r5r6r7r8r9rrorTrr?rgr r rrerBrrs@rDrrs!/!/3 ! ((! #5<<#=>!  t+ ! +, !  ! ! rCrcz\rSrSr%\\S'SrS/rSrSr Sr Sr Sr Sr \\S.r\R$"5S5rS rg ) MLCDPreTrainedModeli rk vision_modelrTF)r attentionsc P URRn[U[5(aURRn[R "UR SURS-U-S9 [R "URRURRU-S9 [R"UR[R"URRS5R!S55 g[U["5(aURRnURS-SURR$-S--U-nURS-U-n[R "UR&RUS9 [R "UR(RUS9 [R "UR*RUS9 [R "UR,RUS9 g[U[.5(aURRnURR0S-SURR$-S--U-nSURR0-S-U-n[R "UR2RUS9 [R "UR4RUS9 g[U[65(amURRnURR0URR8-S-S-U-n[R "UR:SUS9 g[U[<R>5(aA[R@"URB5 [RD"UR5 g[U[<RF5(a.URBb![R@"URB5 g[U[H5(anS URJ[R"S URLS[RNS 9URL- -- n[R"URPU5 gg) zInitialize the weightsr/g)meanstd)rrP)r&rPrNr2rrv))rkr3 isinstanceriinitnormal_r| embed_dimrxryr1copy_ position_idsrTrUrwrXrr#rrrrrFr fc1fc2MLCDVisionModelr% class_pos_embnn LayerNormzeros_biasones_LinearrIthetarRrArV)r]modulefactor in_proj_std out_proj_stdfc_std pos_emb_stdrVs rD _init_weights!MLCDPreTrainedModel._init_weights0s7// f2 3 3[[33F LL//cv?O?OQU?UX^?^ _ LL//66FMM<[<[^d[TU]U5 [URUR-S-5Ul[ R"[R"SURUR-S-55Ul g)Nrr&) rnrorIr r%vision_rotary_embeddingr ParameterrTrandnrrqs rDroMLCDVisionModel.__init__Vsh  ':6;M;MQWQkQk;kop;p'q$\\%++a9K9KvOiOi9imn9n*oprCNrtrrMc Uc [S5eURSURR-nURSURR-nUR X45nUR UR R5n[R"UR U/SS9n[R"XU4SS9nUR5UR54nURU5nURU5nUR"S UUS.UD6n U Sn U SS2SSS24n URU 5n [!U U S9$) ac Example: ```python >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> from transformers import AutoProcessor, MLCDVisionModel >>> model = MLCDVisionModel.from_pretrained("DeepGlint-AI/mlcd-vit-bigG-patch14-448") >>> processor = AutoProcessor.from_pretrained("DeepGlint-AI/mlcd-vit-bigG-patch14-448") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs, output_attentions=True) >>> features = outputs.last_hidden_state >>> print(f"Extracted features shape: {features.shape}") >>> print(f"Number of attention layers: {len(outputs.attentions)}") >>> print(f"Attention shape: {outputs.attentions[0].shape}") ```Nz You have to specify pixel_valuesrPrrQ)rr)r pooler_outputr4) ValueErrorrwrkr+rrzrrOrTr}rrr pre_layrnormencoderpost_layernormr ) r]rtrrKrLrdembrrencoder_outputsr pooled_outputs rDreMLCDVisionModel.forward[sI:  ?@ @)//3t{{7M7MM(..r2dkk6L6LL556H\'**4+=+=+D+DED$6$6#GQOii8bA"wwy#'')4 5 ))-8 ,, ' 3   ,A.)!Q'2 ++M: )/'  rC)rrrm)r5r6r7r8rrorTrr r r?r rerBrrs@rDrrUsTq/q268 ''$.8 +,8  + + 8 8 rCr)rrr)1collections.abcrrTtorch.nnrhuggingface_hub.dataclassesrrrconfiguration_utilsrmodeling_outputsrr modeling_utilsr r processing_utilsr utilsr rrclip.modeling_cliprrrrrrllama.modeling_llamarqwen2_vl.modeling_qwen2_vlrr get_loggerr5loggerrrFrIrirrrrr__all__r4rCrDrs% .&3KF&@@;[   H %CD#$'#$E#$L g /D/$9)M9)x'''T. +. b12/1212h> o> B rC