Zjw&SrSSKrSSKJr SSKJr SSKrSSKJr SSKJ r SSK J r SS K Jr SS KJrJrJrJr SS KJrJr SS KJr SS KJrJrJrJrJr SSKJ r J!r! SSK"J#r# SSK$J%r% \RL"\'5r("SS\RR5r*"SS\RR5r+SAS\RRS\RXS\RXS\RXS\RXS-S\-S-S\-S\\4Sjjr."SS \RR5r/"S!S"\RR5r0"S#S$\RR5r1"S%S&\RR5r2"S'S(\RR5r3"S)S*\5r4"S+S,\RR5r5\"S-S.\55r6\"S/S0\655r7"S1S2\RR5r8\"S3S49"S5S6\655r9\"S7S49"S8S9\655r:\"S:S49\"S;S<\555r;\"S=S49"S>S?\655r[TU]5 [R"[R "SSUR 55Ul[R"[R "SSUR 55UlU(a6[R"[R "SSUR 55OSUl [U5Ul URRn[R"[R "SUS-UR 55Ul [R"UR5UlUR"Ulg)Nr)super__init__r Parametertorchzeros hidden_size cls_tokendistillation_token mask_tokenDeiTPatchEmbeddingspatch_embeddings num_patchesposition_embeddingsDropouthidden_dropout_probdropout patch_size)selfrrr. __class__s w/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/deit/modeling_deit.pyr$DeiTEmbeddings.__init__0s ekk!Q8J8J&KL"$,,u{{1aASAS/T"UQ_",,u{{1a9K9K'LMei 3F ;++77 #%<< A{QPVPbPb0c#d zz&"<"<=  ++ embeddingsheightwidthcURSS- nURRSS- n[RR 5(dXE:XaX#:Xa UR$URSS2SS24nURSS2SS24nURSnX R -n X0R -n [ US-5n URSXU5nURSSSS5n[RRUX4SS S 9nURSSSS5RSSU5n[R"Xg4SS 9$) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing and 2 class embeddings. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rr"N?rrbicubicF)sizemode align_cornersdim)shaper/r&jit is_tracingr3rreshapepermuter functional interpolateviewcat) r4r9r:r;r. num_positionsclass_and_dist_pos_embedpatch_pos_embedrD new_height new_widthsqrt_num_positionss r6interpolate_pos_encoding'DeiTEmbeddings.interpolate_pos_encoding<sU!&&q)A- 0066q9A= yy##%%+*F6?++ +#'#;#;ArrE#B 221ab59r". __, &}c'9:)11!5G]`a)11!Q1=--33 ( 4 *11!Q1=BB1b#Nyy2D!LLr8 pixel_valuesbool_masked_posrTcURu pEnURU5nUR5upnUbIURR XS5n UR S5R U 5n USU - -X--nURR USS5n URR USS5n [R"XU4SS9nURnU(aURXuU5nX~-nURU5nU$)Nr=g?rrC)rEr-r@r+expand unsqueezetype_asr)r*r&rMr/rTr2)r4rVrWrT_r:r;r9 batch_size seq_length mask_tokensmask cls_tokensdistillation_tokensposition_embeddings r6forwardDeiTEmbeddings.forwardds  +001e**<8 $.OO$5!   &//00LK",,R088ED#sTz2[5GGJ^^**:r2> "55<!>zSX!Y 4 \\*- r8)r)r*r2r+r-r3r/)FNF)__name__ __module__ __qualname____firstlineno____doc__rboolr$r&TensorintrT BoolTensorrd__static_attributes__ __classcell__r5s@r6rr+s ,z ,4 ,D , ,&M5<<&M&MUX&M]b]i]i&MV48). ll))D0#'   r8rcf^\rSrSrSrU4SjrS\RS\R4SjrSr U=r $)r,z This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. c>[TU]5 URURp2URUR pT[ U[RR5(aUOX"4n[ U[RR5(aUOX34nUSUS-USUS--nX lX0lX@lX`l [R"XEX3S9Ul g)Nrr) kernel_sizestride)r#r$ image_sizer3 num_channelsr( isinstance collectionsabcIterabler.rConv2d projection)r4rrxr3ryr(r.r5s r6r$DeiTPatchEmbeddings.__init__s !'!2!2F4E4EJ$*$7$79K9Kk#-j+//:R:R#S#SZZdYq #-j+//:R:R#S#SZZdYq !!} 1 5*Q-:VW=:XY $$(&))L:ir8rVr cURup#pEX0R:wa [S5eURU5R S5R SS5nU$)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.r"r)rEry ValueErrorrflatten transpose)r4rVr]ryr:r;xs r6rdDeiTPatchEmbeddings.forwards[2>2D2D/ & ,, ,w  OOL ) 1 1! 4 > >q! Dr8)rxryr.r3r) rgrhrirjrkr$r&rmrdrprqrrs@r6r,r,s. jELLU\\r8r,modulequerykeyvalueattention_maskscalingr2kwargsc UcURS5S-n[R"XRSS55U-nUbX-n[R R USS9n[R RXURS9n[R"X5n U RSS5R5n X4$)Nr=r"rrC)ptrainingr) r@r&matmulrrrJsoftmaxr2r contiguous) rrrrrrr2r attn_weights attn_outputs r6eager_attention_forwardrs**R.D(<<}}Q':;gEL!#4 ==((2(>L==((6??([L,,|3K''1-88:K  $$r8c^\rSrSrS\4U4SjjrS\RS\\ S\ \R\R44Sjr Sr U=r $) DeiTSelfAttentionrc0>[TU]5 URUR-S:wa7[ US5(d&[ SURSURS35eXlURUl[URUR- 5UlURUR-Ul URUl URS-Ul SUl [R"URURUR S9Ul[R"URURUR S9Ul[R"URURUR S9Ulg) Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .rF)bias)r#r$r(num_attention_headshasattrrrrnattention_head_size all_head_sizeattention_probs_dropout_prob dropout_probr is_causalrLinearqkv_biasrrrr4rr5s r6r$DeiTSelfAttention.__init__sG     : : :a ?PVXhHiHi"6#5#5"67334A7   #)#=#= #&v'9'9FV>VV HH]+00)<FFq!L jj/44i@JJ1aP jj/44i@JJ1aP (?(M(M KK , ,.E) *=      * nnLL#}}C$2C2C *  * & #0"4"4"6s";t?Q?Q>S"S%--.EF --r8) rrrrrrrrrr)rgrhrirjrr$r&rmrrtuplerdrprqrrs@r6rrsS]z](.||.+,. u||U\\) * ..r8rc^\rSrSrSrS\4U4SjjrS\RS\RS\R4Sjr S r U=r $) DeiTSelfOutputz The residual connection is defined in DeiTLayer instead of here (as is the case with other models), due to the layernorm applied before each block. rc>[TU]5 [R"URUR5Ul[R "UR5UlgN) r#r$rrr(denser0r1r2rs r6r$DeiTSelfOutput.__init__sB YYv1163E3EF zz&"<"<= r8r input_tensorr cJURU5nURU5nU$rrr2r4rrs r6rdDeiTSelfOutput.forwards$ =1  ]3 r8r) rgrhrirjrkrr$r&rmrdrprqrrs@r6rrsB >z> U\\RWR^R^r8rct^\rSrSrS\4U4SjjrS\RS\\ S\R4Sjr Sr U=r $) DeiTAttentioni rcb>[TU]5 [U5Ul[ U5Ulgr)r#r$r attentionroutputrs r6r$DeiTAttention.__init__ s& *62$V, r8rrr c RUR"U40UD6up4URX15nU$rrr)r4rrself_attn_outputr\rs r6rdDeiTAttention.forwards/ #nn]EfE-= r8r) rgrhrirjrr$r&rmrrrdrprqrrs@r6rr sC-z- ||+,  r8rcj^\rSrSrS\4U4SjjrS\RS\R4SjrSr U=r $)DeiTIntermediateirc>[TU]5 [R"URUR 5Ul[UR[5(a[URUl gURUl gr) r#r$rrr(intermediate_sizerrz hidden_actstrrintermediate_act_fnrs r6r$DeiTIntermediate.__init__s` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $r8rr cJURU5nURU5nU$rrr)r4rs r6rdDeiTIntermediate.forward&s& =1 00? r8r rgrhrirjrr$r&rmrdrprqrrs@r6rrs/9z9U\\ellr8rc^\rSrSrS\4U4SjjrS\RS\RS\R4SjrSr U=r $) DeiTOutputi-rc>[TU]5 [R"URUR 5Ul[R"UR5Ul gr) r#r$rrrr(rr0r1r2rs r6r$DeiTOutput.__init__.sB YYv779K9KL zz&"<"<= r8rrr cRURU5nURU5nX-nU$rrrs r6rdDeiTOutput.forward3s, =1  ]3 %4 r8rrrrs@r6rr-s=>z> U\\RWR^R^r8rcx^\rSrSrSrS\4U4SjjrS\RS\ \ S\R4Sjr S r U=r $) DeiTLayeri;z?This corresponds to the Block class in the timm implementation.rcj>[TU]5 URUlSUl[ U5Ul[ U5Ul[U5Ul [R"URURS9Ul[R"URURS9Ulg)Nreps)r#r$chunk_size_feed_forward seq_len_dimrrr intermediaterrr LayerNormr(layer_norm_epslayernorm_beforelayernorm_afterrs r6r$DeiTLayer.__init__>s '-'E'E$&v.,V4 ( " V-?-?VEZEZ [!||F,>,>FDYDYZr8rrr c URU5nUR"U40UD6nXA-nURU5nURU5nUR XQ5nU$r)rrrrr)r4rrhidden_states_normattention_output layer_outputs r6rdDeiTLayer.forwardHsl "22=A>>*<GG)8 ++M: ((6 {{<? r8)rrrrrrr)rgrhrirjrkrr$r&rmrrrdrprqrrs@r6rr;sHI[z[||+,  r8rc`^\rSrSrS\4U4SjjrS\RS\\ S\ 4Sjr Sr U=r $) DeiTEncoderi^rc>[TU]5 Xl[R"[ UR 5Vs/sHn[U5PM sn5UlSUl gs snfrf) r#r$rr ModuleListrangenum_hidden_layersrlayergradient_checkpointing)r4rr\r5s r6r$DeiTEncoder.__init___sR  ]]uVE]E]?^#_?^!If$5?^#_` &+#$`sA&rrr c LURH nU"U40UD6nM [US9$)N)last_hidden_state)rr )r4rr layer_modules r6rdDeiTEncoder.forwardes. !JJL(A&AM'??r8)rrr)rgrhrirjrr$r&rmrrr rdrprqrrs@r6rr^sD,z, @||@+,@  @@r8rc\rSrSr%\\S'SrSrSrSr S/r Sr Sr Sr Sr\\S.r\R&"5S \R*\R,-\R.-S S 4S j5rS rg )DeiTPreTrainedModeliprdeitrV)imageTr)r attentionsrr Nc2[U[R[R-5(ac[R "UR SURRS9 URb![R"UR5 gg[U[R5(aA[R"UR5 [R"UR 5 g[U[5(a[R"UR5 [R"UR5 [R"UR 5 UR"b![R"UR"5 ggg)zInitialize the weightsr)meanstdN)rzrrr~init trunc_normal_weightrinitializer_rangerzeros_rones_rr)r/r*r+)r4rs r6 _init_weights!DeiTPreTrainedModel._init_weightss fbii"))3 4 4   v}}3DKK[TU]U5 Xl[XS9Ul[ U5Ul[R"URURS9Ul U(a [U5OSUl UR5 g)z add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer use_mask_token (`bool`, *optional*, defaults to `False`): Whether to use a mask token for masked image modeling. )rrN)r#r$rrr9rencoderrrr(r layernorm DeiTPoolerpooler post_init)r4rr rr5s r6r$DeiTModel.__init__si   (O"6* f&8&8f>S>ST,=j(4  r8c.URR$r)r9r-)r4s r6get_input_embeddingsDeiTModel.get_input_embeddingss///r8)tie_last_hidden_statesrVrWrTrc Uc [S5eURRRRR nUR U:waUR U5nURXUS9nURU5nURnURU5nURbURU5OSn [UU S9$)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Nz You have to specify pixel_valuesrWrT)r pooler_output) rr9r-rr dtypetor"rr#r%r ) r4rVrWrTrexpected_dtypeembedding_outputencoder_outputssequence_output pooled_outputs r6rdDeiTModel.forwards  ?@ @99DDKKQQ    /'??>:L?? Tl+ ,0<<8H+I);;..98< 8O O4UY )-'  r8)rr9r"r#r%)TFNNF)rgrhrirjrrlr$r,r)rrrr&rmrorrr rdrprqrrs@r6rrszd[_lp&0&90 E2-137).  llT)  ))D0  #'  +,  $  3   r8rcj^\rSrSrS\4U4SjjrS\RS\R4SjrSr U=r $)r$irc>[TU]5 [R"URUR 5Ul[URUl gr) r#r$rrr(pooler_output_sizerr pooler_act activationrs r6r$DeiTPooler.__init__s> YYv1163L3LM  !2!23r8rr c\USS2S4nURU5nURU5nU$)Nr)rr<)r4rfirst_token_tensorr5s r6rdDeiTPooler.forwards6+1a40 #56  6 r8)r<rrrrs@r6r$r$s/4z4 U\\ellr8r$ad DeiT Model with a decoder on top for masked image modeling, as proposed in [SimMIM](https://huggingface.co/papers/2111.09886). Note that we provide a script to pre-train this model on custom data in our [examples directory](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining). ) custom_introc ^\rSrSrS\SS4U4Sjjr\\S S\RS-S\RS-S\ S \ \ S\4 S jj55rS rU=r$) DeiTForMaskedImageModelingirr NcH>[TU]U5 [USSS9Ul[R "[R "URURS-UR-SS9[R"UR55Ul UR5 g)NFT)r rr"r) in_channels out_channelsrv) r#r$rrr Sequentialr~r(encoder_stridery PixelShuffledecoderr&rs r6r$#DeiTForMaskedImageModeling.__init__s  fdS }} II"..#22A58K8KK  OOF11 2    r8rVrWrTrc PUR"U4UUS.UD6nURnUSS2SS24nURupxn [US-5=pUR SSS5R XyX5nUR U5n Sn UGbURRURR-nUR SX5nURURRS5RURRS5RS5R5n[RRXSS 9nUU-R!5UR!5S -- URR"- n [%U U UR&UR(S 9$) a bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Examples: ```python >>> from transformers import AutoImageProcessor, DeiTForMaskedImageModeling >>> import torch >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> image_processor = AutoImageProcessor.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> model = DeiTForMaskedImageModeling.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2 >>> pixel_values = image_processor(images=image, return_tensors="pt").pixel_values >>> # create random boolean mask of shape (batch_size, num_patches) >>> bool_masked_pos = torch.randint(low=0, high=2, size=(1, num_patches)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss, reconstructed_pixel_values = outputs.loss, outputs.reconstruction >>> list(reconstructed_pixel_values.shape) [1, 3, 224, 224] ```r-Nrr=r>rr"none) reductiongh㈵>)lossreconstructionrr)rrrErnrIrHrJrrxr3repeat_interleaverZrrrJl1_losssumryr rr)r4rVrWrTroutputsr4r]sequence_lengthryr:r;reconstructed_pixel_valuesmasked_im_lossr@r`reconstruction_losss r6rd"DeiTForMaskedImageModeling.forwardsN/3ii / +%=/  / "33*!QrT'24C4I4I1 \_c122)11!Q:BB:]ck&*\\/%B"  &;;))T[[-C-CCD-55b$EO11$++2H2H!L""4;;#9#91=1  #%--"7"7 lr"7"s 1D8==?488:PTCTUX\XcXcXpXppN(5!//))   r8)rJrr7)rgrhrirjrr$rrr&rmrorlrrr rdrprqrrs@r6rCrCszd"-137). I llT)I ))D0I #' I +, I # I I r8rCz DeiT Model transformer with an image classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for ImageNet. c ^\rSrSrS\SS4U4Sjjr\\S S\RS-S\RS-S\ S \ \ S\ 4 S jj55rS rU=r$) DeiTForImageClassificationiMrr Nc.>[TU]U5 URUl[USS9UlURS:a+[ R "URUR5O[ R"5Ul UR5 gNF)r r) r#r$ num_labelsrrrrr(Identity classifierr&rs r6r$#DeiTForImageClassification.__init__Tss   ++f> OUN_N_bcNc"))F$6$68I8IJikititiv r8rVlabelsrTrc UR"U4SU0UD6nURnURUSS2SSS245nSnUbUR"X'UR40UD6n[ UUUR URS9$)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, DeiTForImageClassification >>> import torch >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> torch.manual_seed(3) # doctest: +IGNORE_RESULT >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> # note: we are loading a DeiTForImageClassificationWithTeacher from the hub here, >>> # so the head will be randomly initialized, hence the predictions will be random >>> image_processor = AutoImageProcessor.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> model = DeiTForImageClassification.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> # model predicts one of the 1000 ImageNet classes >>> predicted_class_idx = logits.argmax(-1).item() >>> print("Predicted class:", model.config.id2label[predicted_class_idx]) Predicted class: Polaroid camera, Polaroid Land camera ```rTNr)rOlogitsrr)rrr` loss_functionrr rr) r4rVrbrTrrTr4rdrOs r6rd"DeiTForImageClassification.forward`sV/3ii / %=/ /  "33Aq!9:  %%fdkkLVLD$!//))   r8)r`rr^r7)rgrhrirjrr$rrr&rmrlrrr rdrprqrrs@r6r[r[Ms z d -1&*). = llT)=  t#= #' = +, =  = = r8r[zC Output type of [`DeiTForImageClassificationWithTeacher`]. c\rSrSr%SrSr\RS-\S'Sr \RS-\S'Sr \RS-\S'Sr \ \RS-\S'Sr \ \RS-\S'S rg) +DeiTForImageClassificationWithTeacherOutputiaF logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores as the average of the cls_logits and distillation logits. cls_logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores of the classification head (i.e. the linear layer on top of the final hidden state of the class token). distillation_logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores of the distillation head (i.e. the linear layer on top of the final hidden state of the distillation token). Nrd cls_logitsdistillation_logitsrrr)rgrhrirjrkrdr& FloatTensorrrirjrrrrprr8r6rhrhs} (,FE   $++/J!!D(/48**T1859M5**+d2926Je''(4/6r8rha DeiT Model transformer with image classification heads on top (a linear layer on top of the final hidden state of the [CLS] token and a linear layer on top of the final hidden state of the distillation token) e.g. for ImageNet. .. warning:: This model supports inference-only. Fine-tuning with distillation (i.e. with a teacher) is not yet supported. c ^\rSrSrS\SS4U4Sjjr\\S S\RS-S\ S\ \ S\ 4S jj55rS rU=r$) %DeiTForImageClassificationWithTeacherirr Nc>[TU]U5 URUl[USS9UlURS:a+[ R "URUR5O[ R"5Ul URS:a+[ R "URUR5O[ R"5Ul UR5 gr]) r#r$r^rrrrr(r_cls_classifierdistillation_classifierr&rs r6r$.DeiTForImageClassificationWithTeacher.__init__s   ++f> AG@Q@QTU@UBIIf((&*;*; <[][f[f[h AG@Q@QTU@UBIIf((&*;*; <[][f[f[h $ r8rVrTrc UR"U4SU0UD6nURnURUSS2SSS245nURUSS2SSS245nXg-S- n[ UUUUR UR S9$)NrTrrr")rdrirjrr)rrrorprhrr) r4rVrTrrTr4rirjrds r6rd-DeiTForImageClassificationWithTeacher.forwards/3ii / %=/ /  "33((Aq)AB "::?1aQR7;ST2a7:! 3!//))   r8)rorrpr^rf)rgrhrirjrr$rrr&rmrlrrrhrdrprqrrs@r6rmrmsozd"-1). llT) #' +,  5   r8rm)r[rmrCrr)Nr)>rkcollections.abcr{r dataclassesrr&rrr  activationsrmodeling_layersr modeling_outputsr r r r modeling_utilsrrprocessing_utilsrutilsrrrrr utils.genericrrutils.output_capturingrconfiguration_deitr get_loggerrgloggerModulerr,rmfloatrrrrrrrrrrr$rCr[rhrm__all__rr8r6rsa$! &!9 G&XXI5*   H %VRYYVr"))N!% II% <<% % << % LL4' % T\ %%'(%:4. 4.pRYY$ BII " ryy   *F@"))@$////D: #: : |    ] !4]  ] @ L !4L  L ^  7+7  7& . ,?.  . b r8