Zj\SrSSKrSSKrSSKJr SSKJr SSKJr SSKrSSKJ r SSK J r SS K Jr SS KJr SS KJrJr SS KJr SS KJr SSKJrJrJrJrJrJr SSKJ r J!r!J"r"J#r#J$r$ \RJ"\&5r'Sr(Sr)Sr*\$\"-\#-r+\\"SS9"SS\555r,\"SS9\"SS\555r-\\"SS9"SS\555r."SS\ R^5r0"SS \ R^5r1"S!S"\ R^5r2"S#S$\ R^5r3"S%S&\ R^5r4"S'S(\ R^5r5"S)S*\ R^5r6"S+S,\ R^5r7"S-S.\5r8"S/S0\ R^5r9"S1S2\ R^5r:\"S3S4\55r;\"S5S6\;55r<\"S7S8\;55r=\"S9S:\;55r>\"S;S<\;55r?"S=S>\ R^5r@"S?S@\ R^5rA"SASB\ R^5rB\"SCS9"SDSE\;55rC"SFSG\ R^5rD"SHSI\ R^5rE"SJSK\ R^5rF"SLSM\ R^5rG\"SNS9"SOSP\;55rH/SQQrIg)RzPyTorch FLAVA model.N) OrderedDict) dataclass)Any)nn)initialization)ACT2FN)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPooling)PreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tuplelogging torch_int) FlavaConfigFlavaImageCodebookConfigFlavaImageConfigFlavaMultimodalConfigFlavaTextConfigzfacebook/flava-image-codebookg$(~k@a Output from FlavaModel containing embeddings and outputs from individual encoders. Note that `image_embeddings` and `text_embeddigns` returned are similar to pooled output returned from a transformer. If you want embeddings for contrastive loss or retrieval use a FLAVA model's `image_projection` and `text_projection` layers on `image_embeddings` and `text_embeddings` respectively. ) custom_introc\rSrSr%SrSr\RS-\S'Sr \ S-\S'Sr \RS-\S'Sr \ S-\S'Sr \RS-\S'Sr\ S-\S 'S \\4S jrS rg) FlavaModelOutput3a image_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `pixel_values` are present): The image embeddings which are basically the pooled output of [`FlavaImageModel`]. image_output (`BaseModelOutputWithPooling`, *optional*, returned when `pixel_values` are present): The output of the [`FlavaImageModel`]. text_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` are present): The text embeddings which are basically the pooled output of [`FlavaTextModel`]. text_output (`BaseModelOutputWithPooling`, *optional*, returned when `input_ids` are present): The output of the [`FlavaTextModel`]. multimodal_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` and `pixel_values` are present and `skip_multimodal_encoder` is `None` or `False`): The multimodal embeddings which are basically the pooled output of [`FlavaTextModel`]. multimodal_output (`BaseModelOutputWithPooling`, returned when `input_ids` and `pixel_values` are present and `skip_multimodal_encoder` is `None` or `False`): The output of the [`FlavaMultimodalModel`]. Nimage_embeddings image_outputtext_embeddings text_outputmultimodal_embeddingsmultimodal_outputreturncJ^[U4SjTR555$)Nc3n># UH*nUS;aTUO[TU5R5v M, g7f))r"r r$Ngetattrto_tuple).0kselfs y/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/flava/modeling_flava.py ,FlavaModelOutput.to_tuple..Us<   TTDGZabfhiZjZsZsZu u s25tuplekeysr-s`r.r*FlavaModelOutput.to_tupleTs# YY[   )__name__ __module__ __qualname____firstlineno____doc__rtorch FloatTensor__annotations__r r r!r"r#r$r2rr*__static_attributes__r7r6r.rr3s 26e''$.56:L,t3:04OU&&-459K+d296:5,,t3:;?1D8? %* r6rz@ Class representing pretraining losses from FLAVA model c\rSrSr%SrSr\RS-\S'Sr \RS-\S'Sr \RS-\S'Sr \RS-\S'Sr \RS-\S'Sr \RS-\S 'S \4S jrS rg) FlavaLosses[as mim (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `mim_labels` and `pixel_values` are present, `input_ids_masked` is absent and `mim_weight` > 0.): Masked Image Modeling loss as used in BeIT calculated only for unimodal image data. mlm (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `mlm_labels` and `input_ids_masked` are present, `pixel_values` is absent and `mlm_weight` > 0.): Masked Language Modeling loss as used in BERT calculated only for unimodal text data. itm (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `itm_labels`, `input_ids_masked`, `pixel_values` are present and `itm_weight` > 0.): Image Text Matching (ITM) loss calculated for paired image-text data. Note that ITM loss is calculated on masked pairs in FLAVA. global_contrastive (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `input_ids` and `pixel_values` are present and `global_contrastive_weight` > 0.): Contrastive loss for image-text similarity similar to CLIP but calculated globally for paired image-text data. This is calculated on unmasked images and texts. mmm_image (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `mim_labels`, `pixel_values` and `input_ids_masked` are present and `mmm_image_weight` > 0.): Masked Multimodal Modeling loss's image component calculated on paired image-text data. mmm_text (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `mlm_labels`, `pixel_values` and `input_ids_masked` are present and `mmm_text_weight` > 0.): Masked Multimodal Modeling loss's text component calculated on paired image-text data. Nmimmlmitmglobal_contrastive mmm_imagemmm_textr%cHSnUR5H nUcMSn U$ U$)NTF)values)r-all_nonevs r.rLFlavaLosses.all_nonezs0A}  r6r7)r8r9r:r;r<rDr=r>r?rErFrGrHrIboolrLr@r7r6r.rBrB[s"%)C  T !($(C  T !($(C  T !(37))D07*.Iu  4'.)-He$&-$r6rBa Output from FlavaForPreTraining containing embeddings, and outputs from individual encoders. Note that `image_embeddings` and `text_embeddings` returned are similar to pooled output returned from a transformer. If you want embeddings for contrastive loss or retrieval use a FLAVA model's `image_projection` and `text_projection` layers on `image_embeddings` and `text_embeddings` respectively. c\rSrSr%SrSr\RS-\S'Sr \ \S'Sr \RS-\S'Sr \ S-\S'Sr\RS-\S'Sr\ S-\S 'Sr\RS-\S 'Sr\ S-\S 'Sr\RS-\S 'Sr\ S-\S 'Sr\RS-\S'Sr\ S-\S'Sr\RS-\S'Sr\ S-\S'Sr\RS-\S'Sr\RS-\S'Sr\RS-\S'Sr\RS-\S'Sr\RS-\S'Sr\RS-\S'Sr\RS-\S'S\\ 4Sjr!Sr"g)FlavaForPreTrainingOutputa loss (`torch.FloatTensor`, *optional*, returned when `return_loss` is True): Total loss calculated for this model. loss_info (`FlavaLosses`): Detailed info for FLAVA Pretraining losses. Check `FlavaLosses` class description for the information on the keys. image_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `pixel_values` are present): The image embeddings which are basically the pooled output of [`FlavaImageModel`]. image_output (`BaseModelOutputWithPooling`, *optional*, returned when `pixel_values` are present): The output of the [`FlavaImageModel`]. text_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` are present): The text embeddings which are basically the pooled output of [`FlavaTextModel`]. text_output (`BaseModelOutputWithPooling`, *optional*, returned when `input_ids` are present): The output of the [`FlavaTextModel`]. multimodal_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` and `pixel_values` are present and `skip_unmasked_multimodal_encoder` is `None` or `False`): The multimodal embeddings which are basically the pooled output of [`FlavaTextModel`]. multimodal_output (`BaseModelOutputWithPooling`, returned when `input_ids` and `pixel_values` are present and `skip_unmasked_multimodal_encoder` is `None` or `False`): The output of the [`FlavaMultimodalModel`]. image_masked_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `pixel_values` are present): The image embeddings which are basically the pooled output of [`FlavaImageModel`]. Uses `bool_masked_pos` to create masked images. image_masked_output (`BaseModelOutputWithPooling`, *optional*, returned when `pixel_values` are present): The output of the [`FlavaImageModel`]. Uses `bool_masked_pos` to create masked images. text_masked_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids_masked` are present): The text embeddings which are basically the pooled output of [`FlavaTextModel`]. text_masked_output (`BaseModelOutputWithPooling`, *optional*, returned when `input_ids_masked` are present): The output of the [`FlavaTextModel`]. multimodal_masked_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` and `pixel_values` are present): The multimodal embeddings which are basically the pooled output of [`FlavaTextModel`]. multimodal_masked_output (`BaseModelOutputWithPooling`, *optional*, returned when `input_ids_masked` and `pixel_values` are present): The output of the [`FlavaMultimodalModel`]. mim_logits (`torch.FloatTensor` of shape `(batch_size, num_image_patches, image_vocab_size)` or of shape `(total_masked_patches, image_vocab_size)` , *optional*, returned when `pixel_values` are present and `input_ids_masked` are not): The logits for MIM unimodal loss. Uses `book_masked_pos` to get masked patches. The flattened output is returned when `bool_masked_pos` has some of the patches masked. mlm_logits (`torch.FloatTensor` of shape `(batch_size, text_seq_length, text_vocab_size)` or of shape `(total_masked_seq_length, text_vocab_size)`, *optional*, returned when `input_ids_masked` are present and `pixel_values` are not): The logits for MLM unimodal loss. The flattened output is returned when `input_ids_masked` has some of the tokens masked. itm_logits (`torch.FloatTensor` of shape `(batch_size, 2)`, *optional*, returned when `input_ids_masked` and `pixel_values` are present): The logits for ITM loss. Note that ITM loss is calculated on masked pairs in FLAVA. contrastive_logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeddings` and `text_embeddings` but passed through FLAVA's `image_projection` and `text_projection` layers respectively. This represents the image-text similarity scores. This is calculated on unmasked images and texts. contrastive_logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeddings` and `image_embeddings` but passed through FLAVA's `text_projection` and `image_projection` layers respectively. This is calculated on unmasked images and texts. mmm_image_logits (`torch.FloatTensor` of shape `(batch_size, num_image_patches, image_vocab_size)` or of shape`(total_masked_patches, image_vocab_size)`, *optional*, returned when `pixel_values` and `input_ids_masked` are present): The logits for MMM image multimodal loss. Uses `book_masked_pos` to get masked patches. The flattened output is returned when `bool_masked_pos` has some of the patches masked. mmm_text_logits (`torch.FloatTensor` of shape `(batch_size, text_seq_length, text_vocab_size)` or of shape `(`(total_masked_seq_length, text_vocab_size)`), *optional*, returned when `pixel_values` and `input_ids_masked` are present): The logits for MMM text multimodal loss. The flattened output is returned when `input_ids_masked` has some of the tokens masked. Nloss loss_inforr r!r"r#r$image_masked_embeddingsimage_masked_outputtext_masked_embeddingstext_masked_outputmultimodal_masked_embeddingsmultimodal_masked_output mim_logits mlm_logits itm_logitscontrastive_logits_per_imagecontrastive_logits_per_textmmm_image_logitsmmm_text_logitsr%cV^^/SQm[UU4SjTR555$)N)r"r r$rXrVrZc3l># UH)oT;aTUO[TU5R5v M+ g7fNr()r+r,r-transformer_outputss r.r/5FlavaForPreTrainingOutput.to_tuple..s4sgrbc)< r?rTrBrr r r!r"r#r$rUrVrWrXrYrZr[r\r]r^r_r`rar2rr*r@r7r6r.rQrQs5n&*D%  d ")!I{!15e''$.56:L,t3:04OU&&-459K+d296:5,,t3:;?1D8?8<U..5<=A3d:A7;E--4;<@2T9@=A %"3"3d":ABF84?F+/J!!D(/+/J!!D(/+/J!!D(/=A %"3"3d":A<@!2!2T!9@15e''$.504OU&&-4 t%* tr6rQc ^\rSrSrSrSS\S\SS4U4SjjjrS\RS \ S \ S\R4S jr SS \RS \RS-S\S\R4Sjjr SrU=r$)FlavaImageEmbeddingszZ Construct the CLS token, position and patch embeddings. Optionally, also the mask token. configuse_mask_tokenr%Nc>[TU]5 U=(d URn[R"[ R "SSUR55UlU(a6[R"[ R "SSUR55OSUl[URURURURS9Ul URRn[R"[ R "SUS-UR55Ul[R "UR"5UlURUl Xlg)Nr) image_size patch_size num_channels embed_dim)super__init__ mask_tokenr Parameterr=zeros hidden_size cls_tokenPatchEmbeddingsrnrorppatch_embeddings num_patchesposition_embeddingsDropouthidden_dropout_probdropoutrk)r-rkrlr{ __class__s r.rsFlavaImageEmbeddings.__init__s '<6+<+<ekk!Q8J8J&KLQ_",,u{{1a9K9K'LMei /((((,,(( !  ++77 #%<< A{QPVPbPb0c#d zz&"<"<=  ++ r6 embeddingsheightwidthcURSS- nURRSS- n[RR 5(dXE:XaX#:Xa UR$URSS2SS24nURSS2SS24nURSnX R -n X0R -n [ US-5n URSXU5nURSSSS5n[RRUX4SS S 9nURSSSS5RSSU5n[R"Xg4SS 9$) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNg?rrbicubicF)sizemode align_cornersdim)shaper|r=jit is_tracingrorreshapepermuter functional interpolateviewcat) r-rrrr{ num_positionsclass_pos_embedpatch_pos_embedr new_height new_widthsqrt_num_positionss r.interpolate_pos_encoding-FlavaImageEmbeddings.interpolate_pos_encodingsS!&&q)A- 0066q9A= yy##%%+*F6?++ +221bqb59221ab59r". __, &}c'9:)11!5G]`a)11!Q1=--33 ( 4 *11!Q1=BB1b#Nyy/;CCr6 pixel_valuesbool_masked_posrcFURupEpgURXS9nUR5upIn Ub~URR XIS5n UR 5S:Xa!UR URS5S5nURS5RU 5n USU - -X--nURR USS5n [R"X4SS9nU(aXRXU5-nOXR-nURU5nU$)N)rrrrg?rr)rrzrrtexpandrr unsqueezetype_asrxr=rrr|r)r-rrr batch_sizerprrrseq_len_ mask_tokensmask cls_tokenss r.forwardFlavaImageEmbeddings.forward)s 3?2D2D/ &**<*k !+!2 Q  &//00bIK""$)"1"6"67K7KA7NPR"S",,R088ED#sTz2[5GGJ^^**:r2> YY 7Q?  $#&C&CJX]&^^J#&>&>>J\\*- r6)rxrkrrtrzror|FNF)r8r9r:r;r<rrOrsr=Tensorintr BoolTensorrr@ __classcell__rs@r.riris/RV&&D5<<&D&DUX&D]b]i]i&DV48). ll))D0#'   r6ric ^\rSrSrSrS S\\\-\\\4-S\\\\4-S\S\4U4SjjjrSS\ RS \ S \ R4S jjr S r U=r$)ryiMz Image to Patch Embedding. rnrorprqcX>[TU]5 [U[RR 5(dX4n[U[RR 5(dX"4nUSUS-USUS--nXlX lXPl[R"X4X"S9Ul g)Nrr) kernel_sizestride) rrrs isinstance collectionsabcIterablernror{rConv2d projection)r-rnrorprqr{rs r.rsPatchEmbeddings.__init__Rs *koo&>&>??$1J*koo&>&>??$1J!!} 1 5*Q-:VW=:XY $$&))Lgr6rrr%c >URup4pVU(dVXPRS:wdX`RS:wa2[SUSUSURSSURSS3 5eURU5R S5R SS5nU$)NrrzInput image size (*z) doesn't match model (z).r)rrn ValueErrorrflatten transpose)r-rrrrprrxs r.rPatchEmbeddings.forwardes2>2D2D/ &'++u8J/J (%9+,Adooa.@-AE OOL ) 1 1! 4 > >q! Dr6)rnr{ror)rir)r8r9r:r;r<rlistr2rsr=rrOrr@rrs@r.ryryMs 9<,. h$s)OeCHo5h%S/)h h  hh& ELL D ]b]i]i  r6ryc^\rSrSrSrU4SjrS S\RS-S\RS-S\RS-4SjjrS r U=r $) FlavaTextEmbeddingsiqzGConstruct the embeddings from word, position and token_type embeddings.c >[TU]5 [R"URUR UR S9Ul[R"URUR 5Ul [R"URUR 5Ul [R"UR URS9Ul [R"UR5UlUR#S[$R&"UR5R)S5SS9 UR#S[$R*"UR,R/5[$R0S9SS9 g) N) padding_idxeps position_idsrrF) persistenttoken_type_ids)dtype)rrrsr Embedding vocab_sizerw pad_token_idword_embeddingsmax_position_embeddingsr|type_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsr}r~rregister_bufferr=arangerrvrrlongr-rkrs r.rsFlavaTextEmbeddings.__init__ts !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<=   ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  r6N input_idsrrc UR5nUSnUcURSS2SU24nUcv[US5(a-URSS2SU24nUR USU5nUnO8[ R "U[ RURRS9nURU5nURU5n X-n URU5n X- n URU 5n URU 5n U $)Nrrr)rdevice)rrhasattrrrr=rvrrrrr|rr) r-rrr input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expanded inputs_embedsrrr|s r.rFlavaTextEmbeddings.forwards   nn&  ^  ,,Q  ^ J": "66|D) ^^J/ \\*- r6)rrr|rrNNN) r8r9r:r;r<rsr=rrr@rrs@r.rrqsWQ $*..2,0 <<$&  t+ llT)  r6rc ^\rSrSrS\SS4U4SjjrS S\RS\RS-S\S\ \R\R4\ \R-4S jjr S r U=r $) FlavaSelfAttentionirkr%Nc>[TU]5 URUR-S:wa7[ US5(d&[ SURSURS35eURUl[ URUR- 5UlURUR-Ul[R"URURURS9Ul [R"URURURS9Ul [R"URURURS9Ul[R"UR 5Ulg)Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .bias)rrrsrwnum_attention_headsrrrattention_head_size all_head_sizerLinearqkv_biasquerykeyvaluer}attention_probs_dropout_probrrs r.rsFlavaSelfAttention.__init__s1     : : :a ?PVXhHiHi"6#5#5"67334A7  $*#=#= #&v'9'9FHHAN jj/44\BLLQPQR !<< 5H5HR5PQ+dii8P8P.QQ  %/@ --//0@b/I,,7 _B %--aAq9DDF "/"4"4"6s";t?Q?Q>S"S%**,CD 6G=2O\M]r6)rrrrrrrr r8r9r:r;FlavaPossibleConfigsrsr=rrOr2rr@rrs@r.rrsG3GG*/3"' #||# t+# # u||U\\) *U5<<-@ @ ##r6rc^\rSrSrSrS\SS4U4SjjrS\RS\RS\R4S jr S r U=r $) FlavaSelfOutputiz The residual connection is defined in FlavaLayer (same as ViTLayer) instead of here (as is the case with other models), due to the layernorm applied before each block. rkr%Nc>[TU]5 [R"URUR5Ul[R "UR5Ulgrd) rrrsrrrwdenser}r~rrs r.rsFlavaSelfOutput.__init__sB YYv1163E3EF zz&"<"<= r6r input_tensorcJURU5nURU5nU$rdrrr-rrs r.rFlavaSelfOutput.forwards$ =1  ]3 r6r) r8r9r:r;r<rrsr=rrr@rrs@r.rrsJ >3>> U\\RWR^R^r6rc ^\rSrSrS\SS4U4SjjrS S\RS\RS-S\S\ \R\R4\ \R-4S jjr S r U=r $) FlavaAttentionirkr%Ncb>[TU]5 [U5Ul[ U5Ulgrd)rrrsr attentionroutputrs r.rsFlavaAttention.__init__s& +F3%f- r6rrrcbURXUS9nURUSU5nU4USS-nU$N)rrrrr!r")r-rrr self_outputsattention_outputrs r.rFlavaAttention.forwardsM ~~ L]&  ;;|A F#% QR(88r6r&rrrs@r.rrs|.3../3"' ||  t+   u||U\\) *U5<<-@ @  r6rcn^\rSrSrS\SS4U4SjjrS\RS\R4SjrSr U=r $) FlavaIntermediateirkr%Nc>[TU]5 [R"URUR 5Ul[UR[5(a[URUl gURUl grd) rrrsrrrwintermediate_sizerr hidden_actstrr intermediate_act_fnrs r.rsFlavaIntermediate.__init__ s` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $r6rcJURU5nURU5nU$rdrr0r-rs r.rFlavaIntermediate.forwards& =1 00? r6r3 r8r9r:r;rrsr=rrr@rrs@r.r+r+s79399U\\ellr6r+c^\rSrSrS\SS4U4SjjrS\RS\RS\R4SjrS r U=r $) FlavaOutputirkr%Nc>[TU]5 [R"URUR 5Ul[R"UR5Ul grd) rrrsrrr-rwrr}r~rrs r.rsFlavaOutput.__init__sB YYv779K9KL zz&"<"<= r6rrcRURU5nURU5nX-nU$rdrrs r.rFlavaOutput.forward s, =1  ]3 %4 r6rr6rs@r.r8r8sE>3>> U\\RWR^R^r6r8c ^\rSrSrSrS\SS4U4SjjrS S\RS\RS-S \ S\ \R\R4\ \R-4S jjr S r U=r $) FlavaLayeri)z?This corresponds to the Block class in the timm implementation.rkr%Ncj>[TU]5 URUlSUl[ U5Ul[ U5Ul[U5Ul [R"URURS9Ul[R"URURS9Ulg)Nrr)rrrschunk_size_feed_forward seq_len_dimrr!r+ intermediater8r"rrrwrlayernorm_beforelayernorm_afterrs r.rsFlavaLayer.__init__,s '-'E'E$'/-f5!&) !# V-?-?VEZEZ [!||F,>,>FDYDYZr6rrrcURURU5UUS9nUSnUSSnXQ-nURU5nURU5nUR Xq5nU4U-nU$r%)r!rCrDrBr")r-rrrself_attention_outputsr(r layer_outputs r.rFlavaLayer.forward8s "&  ! !- 0)/"0"  2!4(,)8 ++M: ((6 {{<? /G+r6)r!r@rBrDrCr"rAr)r8r9r:r;r<rrsr=rrOr2rr@rrs@r.r>r>)sI [3 [ [/3"' || t+  u||U\\) *U5<<-@ @ r6r>c^\rSrSrS\SS4U4SjjrS S\RS\RS-S\S \S \S\ \ -4 S jjr S r U=r $) FlavaEncoderiUrkr%Nc>[TU]5 Xl[R"[ UR 5Vs/sHn[U5PM sn5UlSUl gs snfr) rrrsrkr ModuleListrangenum_hidden_layersr>layergradient_checkpointing)r-rkrrs r.rsFlavaEncoder.__init__VsR  ]]fF^F^@_#`@_1Jv$6@_#`a &+#$asA&rrroutput_hidden_states return_dictc"U(aSOSnU(aSOSn[UR5H0upU(aXa4-nU "XU5n U SnU(dM(XzS4-nM2 U(aXa4-nU(d[SXU455$[XUS9$)Nr7rrc3.# UH ocMUv M g7frdr7)r+rMs r.r/'FlavaEncoder.forward..vsm$[q$[ )last_hidden_stater attentions) enumeraterPr2r ) r-rrrrSrTall_hidden_statesall_self_attentionsi layer_module layer_outputss r.rFlavaEncoder.forward\s#7BD$5b4(4OA#$58H$H!(HYZM)!,M  &91=M[TU]5 [R"URUR5Ul[R "5Ulgrd)rrrsrrrwrTanh activationrs r.rsFlavaPooler.__init__}s9 YYv1163E3EF '')r6rc\USS2S4nURU5nURU5nU$Nr)rrf)r-rfirst_token_tensor pooled_outputs r.rFlavaPooler.forwards6+1a40 #56  6 r6)rfrr6rs@r.rcrc|s%$3$ U\\r6rcc^\rSrSr%\\S'SrSrSr\ R"5S\ R\ R-\ R-SS4U4S jj5rS rU=r$) FlavaPreTrainedModelirkflava)imagetextTmoduler%Nc>[TU]U5 [U[5(a![R "UR 5 g[U[5(ao[R "UR5 [R "UR5 URb![R "UR5 gg[U[5(a|[R"UR[R"URR S5R#S55 [R "UR$5 g[U[&5(a3UR((a![R "UR5 gg[U[*5(a6[R,"UR.UR0R25 gg)zInitialize the weightsNrr)rr _init_weightsrFlavaMaskedPredictionHeadinitzeros_rrirxr|rtrcopy_rr=rrrrFlavaMultimodalModel use_cls_token FlavaModel constant_ logit_scalerklogit_scale_init_value)r-rrrs r.rt"FlavaPreTrainedModel._init_weightss> f% f7 8 8 KK $  4 5 5 KK(( ) KK22 3  , F--.-  3 4 4 JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h KK-- .  4 5 5## F,,-$  + + NN6--t{{/Q/Q R,r6r7)r8r9r:r;rr?base_model_prefixinput_modalitiessupports_gradient_checkpointingr=no_gradrrrrrtr@rrs@r.rnrns[ (&*# ]]_SBII $9BLL$HSTSSr6rncF^\rSrSr%\\S'SrSrSrSS\S\ 4U4Sjjjr S\ R4S jr S \ R4S jr\SS\R"S -S \R$S -S\ S -S\R"S -S\ S -S\ S -S\ S -S\\-4Sjj5rSrU=r$)FlavaImageModelirkzflava.image_modelrrpadd_pooling_layerc>[TU]U5 Xl[U5Ul[ U5Ul[R"URURS9Ul U(a [U5OSUl UR5 g^ add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer rN)rrrsrkrirrKencoderrrrwr layernormrcpooler post_initr-rkrrs r.rsFlavaImageModel.__init__sg   .v6#F+ f&8&8f>S>ST->k&)D  r6r%c.URR$rdrrzr4s r.get_input_embeddings$FlavaImageModel.get_input_embeddingss///r6rc$XRlgrdrr-rs r.set_input_embeddings$FlavaImageModel.set_input_embeddingss +0(r6NrrrrrSrTc UbUOURRnUbUOURRnUbUOURRnUc [ S5eUR XUS9n UR U UUUUS9n U Sn URU 5n URbURU 5OSn U(d X4U SS-$[U U U RU RS9$)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, image_num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Nz You have to specify pixel_values)rrrrrSrTrrrY pooler_outputrrZ) rkrrSrTrrrrrr rrZ) r-rrrrrrSrTkwargsembedding_outputencoder_outputssequence_outputrks r.rFlavaImageModel.forwards 2C1N-TXT_T_TqTq$8$D $++JjJj &1%[TU]U5 Xl[U5Ul[ U5Ul[R"URURS9Ul U(a [U5OSUl UR5 gr)rrrsrkrrrKrrrrwrrrcrrrs r.rsFlavaTextModel.__init__sg   -f5#F+ f&8&8f>S>ST->k&)D  r6r%c.URR$rdrrr4s r.r#FlavaTextModel.get_input_embeddingss...r6rc$XRlgrdrrs r.r#FlavaTextModel.set_input_embeddingss */'r6NrrrrrrSrTc ZUbUOURRnUbUOURRnUbUOURRnUc [ S5eUR 5n Uc[ R"XRS9nURUU 5n URUUUS9n URU U UUUS9n U Sn URU 5n URbURU 5OSnU(d X4U SS-$[U UU RU R S9$) a input_ids (`torch.LongTensor` of shape `(batch_size, text_seq_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, text_seq_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) NzYou have to specify input_idsr)rrrrrrr)rkrrSrTrrr=onesrget_extended_attention_maskrrrrr rrZ)r-rrrrrrSrTrrextended_attention_maskrrrrks r.rFlavaTextModel.forwardsU02C1N-TXT_T_TqTq$8$D $++JjJj &1%[TU]U5 XlURRUlUR(a;[R "[ R"SSUR55Ul [U5Ul [R"URURS9UlU(a [U5OSUlUR#5 g)rrrN)rrrsrkrzrrur=rvrwrxrKrrrrrcrrrs r.rsFlavaMultimodalModel.__init__ds   ![[66   \\%++aFS>ST->k&)D  r6NrrrSrTr%c UbUOURRnUbUOURRnUbUOURRnUR 5upxn UR (a8UR RUSS5n [R"X4SS9nUS- nUc [R"Xx4URS9nURUXx45n URUU UUUS9n U Sn URU 5n URbURU 5OSnU(d X4U SS-$[!U UU R"U R$S9$) z hidden_states (`torch.FloatTensor` of shape `(batch_size, image_num_patches + text_seq_len, hidden_size)`): The concatenated hidden states of unimodal encoders. Nrrrrrrr)rkrrSrTrrzrxrr=rrrrrrrr rrZ)r-rrrrSrTrrrrrrrrrks r.rFlavaMultimodalModel.forwardvsp2C1N-TXT_T_TqTq$8$D $++JjJj &1%[TU]U5 [UR[5(d"[ S[ UR5S35e[UR[5(d"[ S[ UR5S35e[UR[5(d%[ SS[ UR5S3-5eURnURnURnURUl URUl URUlURUl[!U5Ul[%U5Ul[)U5Ul[,R."URUR5Ul[,R."URUR5Ul[,R4"[6R8"UR:R<55Ul[,R."URUR5Ul [,R."URUR5Ul!URE5 g)NzLconfig.text_config is expected to be of type FlavaTextConfig but is of type rzNconfig.image_config is expected to be of type FlavaImageConfig but is of type zMconfig.multimodal_config is expected to be of type FlavaMultimodalConfig but z is of type )#rrrsr text_configr TypeErrortype image_configrmultimodal_configrprojection_dimrwtext_hidden_sizeimage_hidden_sizemm_hidden_sizer text_modelr image_modelrymultimodal_modelrrimage_projectiontext_projectionrur=tensorrkr~r}image_to_mm_projectiontext_to_mm_projectionr)r-rkrrrrs r.rsFlavaModel.__init__s  &,,o>>++,-Q0  &--/?@@,,-.a1  &224IJJ_V%=%= >?qAB  (( ** "44$33 + 7 7!-!9!9/;;(5*<8 45F G " $*@*@$BUBU V!yy)>)>@S@ST<< T[[5W5W(XY&(ii0F0FH[H[&\#%'YYt/D/DdFYFY%Z" r6Nrrrrrr%c zUR"SUUUUSS.UD6nURnURU5UlU$)aQ input_ids (`torch.LongTensor` of shape `(batch_size, text_seq_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, text_seq_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) Examples: ```python >>> import torch >>> from transformers import AutoProcessor, FlavaModel >>> model = FlavaModel.from_pretrained("{0}") >>> processor = AutoProcessor.from_pretrained("{0}") >>> inputs = processor( ... text=["a photo of a cat", "a photo of a dog"], max_length=77, padding="max_length", return_tensors="pt" ... ) >>> with torch.inference_mode(): ... text_features = model.get_text_features(**inputs) ``` T)rrrrrTr7)rrYrr)r-rrrrr text_outputsrYs r.get_text_featuresFlavaModel.get_text_featuressYL48??4 ))% 4  4 )::%)%9%9:K%L "r6rrrc zUR"SUUUUSS.UD6nURnURU5UlU$)a bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, image_num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Examples: ```python >>> import torch >>> from transformers import AutoProcessor, FlavaModel >>> from transformers.image_utils import load_image >>> model = FlavaModel.from_pretrained("{0}") >>> processor = AutoProcessor.from_pretrained("{0}") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = load_image(url) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.inference_mode(): ... image_features = model.get_image_features(**inputs) ``` T)rrrrrTr7)rrYrr)r-rrrrr image_outputsrYs r.get_image_featuresFlavaModel.get_image_featuress[B594D4D5 %+)%= 5  5 *;;&*&;&;>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, FlavaModel >>> model = FlavaModel.from_pretrained("facebook/flava-full") >>> processor = AutoProcessor.from_pretrained("facebook/flava-full") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(text=["a photo of a cat"], images=image, return_tensors="pt", padding=True) >>> outputs = model(**inputs) >>> image_embeddings = outputs.image_embeddings >>> text_embeddings = outputs.text_embeddings >>> multimodal_embeddings = outputs.multimodal_embeddings >>> outputs.image_embeddings.shape torch.Size([1, 197, 768]) >>> text_embeddings.shape torch.Size([1, 7, 768]) >>> multimodal_embeddings.shape torch.Size([1, 205, 768]) ``` NzRFLAVA model requires hidden states to work. Please set `output_hidden_states=True`)rrrrrSrTrrr)rrrrrrSrTrrr)rrT)rr r!r"r#r$)rkrTrrrrrrrrzr=rrrr)r-rrrrrrrrrrSrTrr image_statesimage_mm_projectionr r! text_statestext_mm_projectionr"r#r$rrrattention_mask_imageattention_multimodalmultimodal_inputs r.rFlavaModel.forward?sL&1%;q>[O!%!;!;KO!L  $   */A/MVm))<)B)B& GQ((66qLG',zz*gNaNhNh'i$',yy2F1W]^'_$'+$$yy*=?Q)RXYZ  $ 5 5 1ES^!6! %6a$8 ! %!   -%+#"7/   r6) rrrrr}rrrrrrrr) NNNNNNNNNTN)r8r9r:r;rr?rsrrr=rrrr2r rrrOr LongTensorr>rrr@rrs@r.r{r{s? ){)V/3.2,0 /<</ t+/ t+ / llT) / +, / + +//b4804.2 *ll*))D0*#'+ *  t+ * +, * + +**X.215.2.2/30448/3)-%)#'N ##d*N ''$.N  t+ N  t+ N , N &&-N $llT1N "&N  $;N #N D[N  ! !N N r6r{cn^\rSrSrS\S\4U4SjjrS\RS\R4SjrSr U=r $) FlavaImageCodebookResPathiin_sizeout_sizec >[TU]5 US-n[5n[R"5US'[R "XSSS9US'[R"5US'[R "XDSSS9US'[R"5US '[R "XDSSS9US '[R"5US '[R "XBSS S9US '[R "U5Ulg)Nrelu_1rrrpaddingconv_1relu_2conv_2relu_3conv_3relu_4rconv_4)rrrsrrReLUr Sequentialpath)r-rrrhid_sizerrs r.rs"FlavaImageCodebookResPath.__init__s q=}X7!QOXX81aPXX81aPXX81aPXMM$' r6rr%c$URU5$rdrr-rs r.r!FlavaImageCodebookResPath.forwardsyy|r6r r8r9r:r;rrsr=rrr@rrs@r.rrs6((s( %,,r6rcr^\rSrSrS\S\S\4U4SjjrS\RS\R4SjrS r U=r $) FlavaImageCodebookBlockirr num_layersc >[TU]5 SUS-- UlX:wa[R"XSSS9UlO[R "5Ul[X5Ulg)Nrrrr) rrrs post_gainrrid_pathIdentityrres_path)r-rrrrrs r.rs FlavaImageCodebookBlock.__init__sQ j!m,  99WAqQDL;;=DL1'D r6rr%cbURU5URURU5--$rdrrrrs r.rFlavaImageCodebookBlock.forwards'||A$--2B!BBBr6rrrs@r.rrsE E Es E ECC%,,CCr6rc ~^\rSrSrS S\S\S\S\S\4 U4SjjjrS\RS \R4S jr S r U=r $) FlavaImageCodebookLayerGroupi num_blocksrrruse_poolc0>[TU]5 [5n[U5H5nUS:Xa[ X4U5USUS-3'M [ XDU5USUS-3'M7 U(a[ R "SS9US'[ R"U5Ulg)Nrblock_rr)rpool) rrrsrrNrr MaxPool2drgroup) r-r rrrr blocksr^rs r.rs%FlavaImageCodebookLayerGroup.__init__s z"AAv+B7V`+aAw'(+B8Wa+bAw'( # \\a8F6N]]6* r6rr%c$URU5$rdrrs r.r$FlavaImageCodebookLayerGroup.forwardszz!}r6rr) r8r9r:r;rrOrsr=rrr@rrs@r.r r sR +3 +C +# +QT +`d + +%,,r6r a" The FLAVA's image codebook model inspired from DALL-E's original encoder. Outputs raw hidden states and can be used to generate image tokens for an image based on DALL-E's vocab. Used to generate labels for MIM. Use `get_codebook_indices` to get image tokens for an image. c^\rSrSr%Sr\\S'SrSrSr S\S\ 4U4Sjjr S\ RS \ R4S jrS\ RS \ R4S jrS\ R S \ R4S jrS rU=r$)FlavaImageCodebooki modelrkrrFrc >[TU]U5 XlURUlURUlUR UlUR UlURUlURUR -n[5n[R"5US'[R"SUR -URSSS9US'[5n[R"URSUR -SSS9US '[UR USUR -SUR -5US '[UR USUR -S UR -5US '[UR US UR -S UR -5US'[UR US UR -SUR -SS9US'[R"U5US'[R"U5UlUR5 URR (aUR#5H nSUlM gg)Nrelurrrconvrinputgroup_1rgroup_2rgroup_3F)r group_4r")rrrsrk num_groupsinput_channelsnum_blocks_per_grouprwrrrrrr rrrfreeze parameters requires_grad)r-rkrr output_blocksrparamrs r.rsFlavaImageCodebook.__init__s    ++$33$*$?$?!!-- ++__t'@'@@ # " f " !d.>.>*>]^hi j f))D$7$7T=M=M9M[\fghw8  % %z1t7G7G3GTM]M]I] y9  % %z1t7G7G3GTM]M]I] y9  % %z1t7G7G3GTM]M]I] y9  % %z1t7G7G3GTM]M]I]hm y==7xmmF+   ;;  *&+#+ r6r%cpS[S[S3 URU5n[R"USS9$)NaI Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Codebook pixel values can be obtained using [`AutoImageProcessor`] by passing `return_codebook_pixels=True`. See [`FlavaImageProcessor.__call__`] for details. Examples: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoImageProcessor, FlavaImageCodebook >>> model = FlavaImageCodebook.from_pretrained("E") >>> image_processor = AutoImageProcessor.from_pretrained("a") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = image_processor([image], return_codebook_pixels=True, return_tensors="pt") >>> inputs = dict(pixel_values=inputs.codebook_pixel_values) >>> outputs = model.get_codebook_indices(**inputs) ``` r)axis)_CHECKPOINT_FOR_CODEBOOK_DOCrr=argmaxr-rz_logitss r.get_codebook_indices'FlavaImageCodebook.get_codebook_indicesEsI 9:V8VWCC_B` a  4;;|,||H1--r6cXURU5n[R"SS9"U5$)Nrr)rrSoftmaxr2s r.get_codebook_probs%FlavaImageCodebook.get_codebook_probscs$;;|,zza **r6c 4S[S[S3 [UR5S:wa[SURS35eURSUR:wa([SURSS UR35eUR U5$) NaJ Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Codebook pixel values can be obtained using [`AutoImageProcessor`] by passing `return_codebook_pixels=True`. See [`FlavaImageProcessor.__call__`] for details. Examples: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoImageProcessor, FlavaImageCodebook >>> model = FlavaImageCodebook.from_pretrained("r.a") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = image_processor([image], return_codebook_pixels=True, return_tensors="pt") >>> inputs = dict(pixel_values=inputs.codebook_pixel_values) >>> outputs = model(**inputs) >>> print(outputs.shape) (1, 196) ``` rz input shape z is not 4drz input has z channels but model built for )r0lenrrr%r)r-rrs r.rFlavaImageCodebook.forwardgs 9:V8VWCC_B` a  : |!! "a '|L,>,>+?zJK K   a D$7$7 7z,*<*rr@rrs@r.rr s  $$$O!&+#*,(*,*,X..%,,.<+u||+ +")E$5$5")ELL")")r6rc.^\rSrSrU4SjrSrSrU=r$)FlavaPredictionHeadTransformicp>[TU]5 [R"URUR5Ul[ UR[5(a[URUl OURUl [R"URURS9Ul g)Nr) rrrsrrrwrrr.r/r transform_act_fnrrrs r.rs%FlavaPredictionHeadTransform.__init__s~ YYv1163E3EF f'' - -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STr6clURU5nURU5nURU5nU$rd)rr@rr4s r.r$FlavaPredictionHeadTransform.forwards4 =1 --m< }5 r6)rrr@r8r9r:r;rsrr@rrs@r.r>r>sUr6r>c2^\rSrSrSU4SjjrSrSrU=r$)ruicF>[TU]5 Xl[U5Ul[ R "URURSS9Ul [ R"[R"UR55Ul UbX Rlgg)NTr)rrrsrkr> transformrrrwrdecoderrur=rvrweight)r-rkrIrs r.rs"FlavaMaskedPredictionHead.__init__ss  5f=yy!3!3V5F5FTR LLV->->!?@  "(LL  r6cJURU5nURU5nU$rd)rGrHrs r.r!FlavaMaskedPredictionHead.forwards" NN1  LLOr6)rrkrHrGrdrDrs@r.rurus)r6ruc.^\rSrSrU4SjrSrSrU=r$) FlavaITMHeadic>[TU]5 Xl[U5Ul[ R "URS5Ulg)Nr) rrrsrkrcrrrrwseq_relationshiprs r.rsFlavaITMHead.__init__s8  !&) " &*<*[TU]5 XlURUlgrd)rrrsrkglobal_backprop_contrastivers r.rs#FlavaGlobalContrastiveHead.__init__s!  +1+M+M(r6cD[R"U5n[RR5(a#[RR 5(d6[R "UR S5URS9nU/nU/nGOUR S5n[RR5n UR(ag[RRRRU5n[RRRRU5nO[U 5V s/sHn [R"U5PM nn [U 5V s/sHn [R"U5PM nn [RRXa5 [RRXr5 U[RR5-[R "XRS9-n[R "U5n[R "U5n[R""XR%SS55U-n [R""X&R%SS55U-n XU4$s sn fs sn f)Nrrr)r=exp distributed is_availableis_initializedrrrget_world_sizerWrr all_gatherrN zeros_likeget_rankrrr) r-rr!r} temperaturelabelsimage_embeddings_alltext_embeddings_alllocal_batch_size world_sizerlogits_per_imagelogits_per_texts r.r"FlavaGlobalContrastiveHead.forwardsii ,   --//u7H7H7W7W7Y7Y\\"2"7"7":CSCZCZ[F$4#5 #2"3 /44Q7 **99;J//(-'8'8';';'F'F'Q'QRb'c$&+&7&7&:&:&E&E&P&PQ`&a#SXYcSd'eSda(8(8(ISd$'eSXYcSd&eSdau'7'78H'ISd#&e!!,,-AT!!,,-@R%(9(9(B(B(DDu|| )@)@HF %yy)=>#ii(;< <<(8:W:WXY[\:]^all,,8V8VWXZ[8\]`kk&88(f&es  J6 J)rkrWrDrs@r.rUrUsN 99r6rUzk The FLAVA model for pretraining which outputs losses, embeddings, logits and transformer outputs. c't^\rSrSrSSSSS.rS!S\S \RS-4U4S jjjrS \ R4S jr \ S"S \ RS-S\ RS-S\ RS-S\ RS-S\ RS-S\ RS-S\ RS-S\ RS-S\ RS-S\S-S\ RS-S\ RS-S\ RS-S\S-S\S\S-S\S-S\\ R\-4$Sjj5rS rU=r$)#FlavaForPreTrainingizmmm_text_head.decoder.biaszmim_head.decoder.biaszmlm_head.decoder.biaszmmm_image_head.decoder.bias)zmmm_text_head.biasz mim_head.biasz mlm_head.biaszmmm_image_head.biasNrkimage_codebookcl>[TU]U5 [U5UlX lURc+UR (a[ UR5Ul[UR5Ul [UR5Ul [U5Ul[UR5Ul[UR5Ul[#U5UlURR&UlURR&UlUR,UlUR.UlUR0UlUR2UlUR4UlUR6UlUR8UlUR:UlUR=5 g)z image_codebook ([`nn.Module`]): If passed, the image codebook will be set to this. Otherwise, it will be initialized using the image_codebook_config defined in the config first as the first parameter. N)rrrsr{rorm init_codebookrimage_codebook_configrurmim_headrmlm_headrNitm_headmmm_image_head mmm_text_headrUglobal_contrastive_headrimage_vocab_sizetext_vocab_size mlm_weight mim_weightglobal_contrastive_weightce_ignore_index itm_weightmmm_image_weightmmm_text_weight skip_unmasked_multimodal_encoderr)r-rkrmrs r.rsFlavaForPreTraining.__init__sO  ' ,    &6+?+?"4V5Q5Q"RD 2&2E2EF 1&2D2DE $V, 78K8KL6v7I7IJ'A&'I$ & 3 3 > >%11<< ++ ++)/)I)I&%55 ++ & 7 7%55060W0W- r6rcpUR5S:a!URURS5S5nU$)Nrrr)rrrrs r. _resize_to_2d!FlavaForPreTraining._resize_to_2ds, 557Q;qvvay"%Ar6rinput_ids_maskedrcodebook_pixel_valuesrrrrrr mlm_labels mim_labels itm_labelsrrSrT return_lossr%c UbUOURRnUbUOURRnU bU O URn UcUb[R S5 UnUR UUUUUU U UUSS9 nUR UUUUU UUUSS9 nSnURnURnURnURnURnS=n=n=n=n=n=n n!S=n"=n#=n$n%S=n&=n'n(UcUbKU cHU(aAURc [S5eUc [S5eURRU5n URS:GaUGbUGc Un)U bURU 5n URU5nUR XR#S5'U)SS2U R%S 5*S2SS24n)U R#UR 5n*U U*n+U)U*SS24n)UR'U)5n"U(aX[(R*R-U"R/S UR05U+R/S 55nUUR-nOUR'U)5n"UR2S:aUbUcUn,U bURU 5n U,SS2U R%S 5*S2SS24n,U R#UR 5n*U U*n-U,U*SS24n,UR5U,5n#U(aX[(R*R-U#R/S UR65U-R/S 55nUUR2-nOUR5U,5n#UR8S:aUbUR;U5n&U bU R#S5n.[<R>"U.RA5U.U.RCS/55nU(a/[(R*R-U&U 5n!U!UR8-n!UbUUnU bU Un U b U Un UUnUGb%URDS:GaUn)UR%S 5S - n/U)SS2S S U/-2SS24n)U bURU 5n URU5nUR XR#S5'U R#UR 5n*U U*n+U)U*SS24n)URGU)5n%U(aX[(R*R-U%R/S UR05U+R/S 55nUURD-nOURGU)5n%UbURHS:aUn,U,SS2UR%S 5*S2SS24n,U bURU 5n U R#UR 5n*U U*n-U,U*SS24n,URKU,5n$U(aX[(R*R-U$R/S UR65U-R/S 55nUURH-nOURKU,5n$UGbUGb{URLS:GajUR ROUSS2SSS245n0[(R*RQU0S S 9n0UR RSUSS2SSS245n1[(R*RQU1S S 9n1URT(a8UR RVRXR[[\[^5 URaU1U0UR RV5un'n(n2UbU'Un'U(Un(U2Un2U(aW[(R*R-U'U25n3[(R*R-U(U25n4U3U4-S - n U URL-n [cUUU!U UUS 9n5U(a5U5Re5(d [gSU5Ri555nU(Gd>UURjbURjRm5OSUURnbURnRm5OSURURpbURpRm5OSUURjbURjRm5OSUURnbURnRm5OSUURpbURpRm5OSU"U#U&U'U'U%U$4n6U(aU5Re5(dUU54U6-n6[sSU655$[uS%0SU_SU5_SU_SURj_SU_SURn_SUR_SURp_SU_SURj_SU_SURn_SU_SURp_SU"_SU#_S U&_S!U'_S"U(_S#U%_S$U$_6$)&a input_ids (`torch.LongTensor` of shape `(batch_size, text_seq_len)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) input_ids_masked (`torch.LongTensor` of shape `(batch_size, text_seq_len)`): Indices of input sequence tokens in the vocabulary. These ones are the masked version of the original task to be used with MLM. Indices can be obtained using [`AutoTokenizer`] along with [`DataCollatorForMaskedLanguageModeling`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) codebook_pixel_values (`torch.FloatTensor` of shape `(batch_size, num_image_patches, patch_size, patch_size, 3)`, *optional*): Pixel values for image patches that are used to compute the image codebook labels for masked image modeling. token_type_ids (`torch.LongTensor` of shape `(batch_size, text_seq_len)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, image_num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). image_attention_mask (`torch.FloatTensor` of shape `(batch_size, image_num_patches)`, *optional*): Mask to avoid performing attention on padding token indices specifically for images. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) skip_unmasked_multimodal_encoder (*bool*, *optional*): Skip any calculations for multimodal encoder for unmasked inputs. FLAVA pretraining doesn't need unmasked multimodal embeddings or outputs as of now. mlm_labels (`torch.LongTensor` of shape `(batch_size, text_seq_len)`, *optional*): Labels for computing the left-to-right language and multimodal masked modeling loss (next word prediction). Indices should be in `[-100, 0, ..., text_config.vocab_size - 1]` (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., text_config.vocab_size - 1]`. mim_labels (`torch.LongTensor` of shape `(batch_size, image_num_patches)`, *optional*): Labels for computing the image and multimodal masked modeling loss. Indices should be in `[-100, 0, ..., image_config.vocab_size - 1]`. Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., image_config.vocab_size - 1]`. If not passed, they are generated automatically using the image codebook assigned to the model. By default, it uses [`FlavaImageCodebook`]. See [`FlavaImageCodebook`] to understand how to generate mim_labels. itm_labels (`torch.LongTensor` of shape `(batch_size, 1)`, *optional*): Labels for computing the image-text matching loss. 0 means the pairs don't match and 1 means they match. The pairs with 0 will be skipped for calculation of MMM and global contrastive losses as well. return_loss (`bool`, *optional*, default to None): Whether to return calculated loss or not. Examples: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import FlavaForPreTraining, AutoProcessor >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> model = FlavaForPreTraining.from_pretrained("facebook/flava-full") >>> processor = AutoProcessor.from_pretrained("facebook/flava-full") >>> text = ["a photo of a cat"] >>> inputs = processor( ... images=[image], ... text=text, ... return_masks=True, ... return_codebook_pixels=True, ... padding=True, ... max_length=77, ... return_tensors="pt", ... ) >>> output = model(**inputs) ``` Nz`input_ids_masked` isn't passed which means MLM loss won't be calculated correctlySetting it to `input_ids` so that model can work. Please pass it if this is unintentional. This is usually OKAY if you are doing inference on unmasked text...T) rrrrrrrrrSrT) rrrrrrrrSrTz`return_loss` is set to True but the image codebook is not initialized and no `mim_labels` have been passed. Reinstantiate the model with `init_codebook` set to True or pass in your custom `mim_labels`z`codebook_pixel_value` are required to generate `mim_labels` if loss is expected. Call `AutoProcessor` with `return_codebook_pixels` set to Truerrrrr)rDrErFrGrHrIc3.# UH obUOSv M g7frir7)r+rSs r.r/.FlavaForPreTraining.forward..Os_I^%5T1eGV^GKK KZK/4D:>> >% # .2N2Z!k&&.&; )0$Y"00EEF[\  ??Q #:#FKgKo!8 %!// ; "&"4"4_"E7;7K7K --d34%7JOOA !}}::"D,A,ABDWD\D\]_D` H/H!]]+=>  ??Q #9#EJfJn 6 %!// ; $5a*//!:L9L9NPQ6Q$R! * d.B.B C &0&?#$5mQ6F$G!!]]+<= !}}::"D,@,@ACVC[C[\^C_ H/H!]]+<=  ??Q #?#K'CDJ%&MM!,  ;;y}} 9==RVQWCXY!}}:::zRH/H/;3OPX3Y0)!+H!5J)!+H!5J&5h&?O ( 38M8MPQ8Q!= /44Q7!;I!3Aq1y=7H!4K!L %!// ; "&"4"4_"E7;7K7K --d34 * d.B.B C &0&?#%7 q8H%I"#'#6#67I#J %']]%@%@(--b$2G2GHJ]JbJbceJf&N#d&;&;;N#'#6#67I#J  ( 38L8Lq8P <  1!6L6Q6QRS6T5T5VXY2Y Z %!// ; * d.B.B C &0&?#$5mQ6F$G!"&"4"45F"G$&MM$?$?',,R1E1EFH[H`H`acHd%M"T%9%99M"&"4"45F"G  'O,GDLjLjmnLn!ZZ771a8PQN]]44^4LN"jj99:J1aQR7:STO mm55o25NO}} &&++223HJ_`;?;W;W1G1G< 8 oy ##3H#= "1(";%h/  " ; ;
  • Q>]>]>i#//88:os,&88D$55>>@   +F.<#8#8#:#:  8F88 8(  " . &22  ,  %00  #/"D"D +<< %< !4 @ @ $:  3>> *F &9%J%J " "! ""# $*:% &)8' (.) *,+  r6)r|rorvr{rmrwrsr}rqrzrrryrtr~rurrrxrd)NNNNNNNNNNNNNNTNN)r8r9r:r;_tied_weights_keysrrrrsr=rrrrr>rOr2rQrr@rrs@r.rlrls;00< !{!BII.2.2/304488<*.*.*.)-%)#'#'%o ##d*o  **T1o ''$. o %0047 o  t+ o  t+o ,o &&-o $llT1o +/+o LL4'o LL4'o LL4'o  $;o #!o "D[#o $D[%o ( u|| 8 8)o o r6rl)rlrrr{ryrnr)Jr<rrr dataclassesrtypingrr=rrrv activationsr modeling_layersr modeling_outputsr r modeling_utilsr processing_utilsrutilsrrrrrrconfiguration_flavarrrrr get_loggerr8rr0rrrrrBrQrriryrrrrr+r8r>rKrcrnrrryr{rrr rr>rurNrUrl__all__r7r6r.rs) #! &!9K-&jj   H %>&)99