Zj(SrSSKJr SSKJr SSKJr SSKrSSKJr SSK J r SS K J r SS KJrJr SS KJrJrJr SS KJrJr SS KJr SSKJrJrJrJrJr SSK J!r!J"r" SSK#J$r$ SSK%J&r& SSK'J(r(J)r)J*r*J+r+ SSK,J-r-J.r. SSK/J0r0 SSK1J2r2J3r3J4r4 \*Rj"\65r7Sr8\)"SS9\"SS\555r9\)"SS9\"SS\555r:"SS \Rv5r<"S!S"\Rv5r="S#S$\Rv5r>"S%S&\Rv5r?"S'S(\Rv5r@"S)S*\Rv5rA"S+S,\Rv5rB"S-S.\Rv5rC"S/S0\Rv5rDSdS1\RvS2\RS3\RS4\RS5\RS-S6\FS-S7\FS8\$\(4S9jjrG"S:S;\Rv5rH"S<S=\Rv5rI"S>S?\Rv5rJ"S@SA\Rv5rK"SBSC\5rL"SDSE\Rv5rM"SFSG\Rv5rN\)"SHSI\"55rO"SJSK\O5rP\)"SLS9"SMSN\O55rQ\)"SOS9"SPSQ\O55rR"SRSS\Rv5rS"STSU\Rv5rT"SVSW\Rv5rU\)"SXS9"SYSZ\O55rV\)"S[S9"S\S]\O55rW"S^S_\Rv5rX\)"S`S9"SaSb\O55rY/ScQrZg)ezPyTorch BridgeTower Model) OrderedDict)Callable) dataclassN)nn)CrossEntropyLoss)initialization)ACT2FNQuickGELUActivation)Cache DynamicCacheEncoderDecoderCache)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentionsMaskedLMOutput ModelOutputSequenceClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward)TransformersKwargsauto_docstringlogging torch_int)can_return_tuplemerge_with_config_defaults)capture_outputs)BridgeTowerConfigBridgeTowerTextConfigBridgeTowerVisionConfigRobertaTokenizerz. Output type of [`BridgeTowerModel`]. ) custom_introc\rSrSr%SrSr\RS-\S'Sr \RS-\S'Sr \RS-\S'Sr \ \RS-\S'Sr \ \RS-\S'S rg) BridgeTowerModelOutput2a text_features (`torch.FloatTensor` of shape `(batch_size, text_sequence_length, hidden_size)`): Sequence of hidden-states at the text output of the last layer of the model. image_features (`torch.FloatTensor` of shape `(batch_size, image_sequence_length, hidden_size)`): Sequence of hidden-states at the image output of the last layer of the model. pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size x 2)`): Concatenation of last layer hidden-state of the first token of the text and image sequence (classification token), respectively, after further processing through layers used for auxiliary pretraining tasks. N text_featuresimage_features pooler_output hidden_states attentions)__name__ __module__ __qualname____firstlineno____doc__r+torch FloatTensor__annotations__r,r-r.tupler/__static_attributes__r0څ/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/bridgetower/modeling_bridgetower.pyr)r)2s|/3M5$$t+2/3NE%%,3.2M5$$t+259M5**+d2926Je''(4/6r;r)z> Output type of ['BridgeTowerForContrastiveLearning'] cP\rSrSr%SrSr\RS-\S'Sr \RS-\S'Sr \ \RS-\S'Sr \ \RS-\S'Sr \ \RS-\S'Sr\ \RS-\S 'Sr\ \RS-\S 'S rg) BridgeTowerContrastiveOutputJa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Image-text contrastive loss. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). text_embeds (`torch.FloatTensor)`, *optional*, returned when model is initialized with `with_projection=True`): The text embeddings obtained by applying the projection layer to the pooler_output. image_embeds (`torch.FloatTensor)`, *optional*, returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. cross_embeds (`torch.FloatTensor)`, *optional*, returned when model is initialized with `with_projection=True`): The text-image cross-modal embeddings obtained by applying the projection layer to the pooler_output. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Nlosslogits text_embeds image_embeds cross_embedsr.r/r0)r1r2r3r4r5r@r6r7r8rArBr9rCrDr.r/r:r0r;r<r>r>Js &*D%  d ")'+FE   $+37Ku(()D0748L%))*T1848L%))*T1859M5**+d2926Je''(4/6r;r>c^\rSrSrU4SjrS\R S\R 4SjrS S\R S\R S-4SjjrSr U=r $) BridgeTowerResidualAttentionjc h>[TU]5 [R"URURS-5Ul[R "URURS9Ul[R"[S[R"URURS-54S[54S[R"URS-UR54/55Ul [R "URURS9UlSUlg)N@epsc_fcgeluc_proj)super__init__rMultiheadAttention hidden_sizeattn LayerNormlayer_norm_epsln_1 ModuleDictrLinearr mlpln_2 attn_maskselfconfig __class__s r<rQ%BridgeTowerResidualAttention.__init__ks ))&*<*P>PTV>VW LL!3!39N9NO == RYYv'9'96;M;MPQ;QRS023ryy););a)?ASASTU  LL!3!39N9NO r; hidden_stateattention_maskc Ub(UR[RURS9nURb.URRUR URS9OSUlUR UUUSURUS9S$)NdtypedeviceF) need_weightsr\key_padding_maskr)tor6boolrgr\rfrT)r^rbrcs r< attention&BridgeTowerResidualAttention.attention|s  %+..UZZ H[H[.\N~~) NN  L$6$6|?R?R  S  yy   nn+     r;NcXRURU5U5-nURU5nURR 5H nU"U5nM X1-nU$N)rlrWr[rZvalues)r^rbrcresidual_statelayers r<forward$BridgeTowerResidualAttention.forwardsZ%tyy7NP^(__yy0 XX__&E .L'%4 r;)rTr\rWr[rZro) r1r2r3r4rQr6Tensorrlrsr: __classcell__r`s@r<rFrFjsI" ell ELL "ELL%,,QUBUr;rFcl^\rSrSrU4SjrSS\R S\R S-4SjjrSrU=r $) BridgeTowerTransformerc>[TU]5 URUlURUlUR(aL[ R "[URS- 5Vs/sHn[U5PM sn5Ul OH[ R "[UR5Vs/sHn[U5PM sn5Ul URUl gs snfs snf)Nr") rPrQrSnum_hidden_layersremove_last_layerr ModuleListrangerF resblocks stop_gradientr^r__r`s r<rQBridgeTowerTransformer.__init__s !--!'!9!9  # #]]?DTE[E[^_E_?`a?`!-f5?`aDN ]]?DTE[E[?\]?\!-f5?\]DN$11 b^s -C)6C.Nrbrcc/nURHNnU"X5nUR(a!URUR55 M=URU5 MP U$ro)rrappenddetach)r^rbrcr.blocks r<rsBridgeTowerTransformer.forwardsU ^^E >L!!$$\%8%8%:;$$\2 $ r;)rSr|rrro r1r2r3r4rQr6rursr:rvrws@r<ryrys. 2ELL%,,QUBUr;ryc^\rSrSrS\4U4SjjrS\RS\S\S\R4Sjr S S \RS\R4S jjr S r U=r $) BridgeTowerVisionEmbeddingsr_c>[TU]5 XlURUlUR UlUR Ul[R"[R"UR55Ul [R"URURUR UR SS9UlUR UR -S-UlURS-Ul[R""UR UR5UlUR'S[R("UR 5R+S5SS9 g)NF) in_channels out_channels kernel_sizestridebiasr" position_idsr" persistent)rPrQr_rS embed_dim image_size patch_sizer Parameterr6randnclass_embeddingConv2d num_channelspatch_embedding num_patches num_positions Embeddingposition_embeddingregister_bufferarangeexpandr]s r<rQ$BridgeTowerVisionEmbeddings.__init__s  ++ ++ ++!||EKK,GH!yy++??  !OOt>1D!--1"$,,t/A/A4>>"R ^U\\$:L:L-M-T-TU\-]jopr; embeddingsheightwidthreturncURSS- nURRRS5nURSS- n[R R 5(d%XF:Xa X#:XaURUR5$USS2SS24nUSS2SS24nURSn X R-n X0R-n [US-5n URSXU 5nURSSSS5n[RRUX4SS S 9nURSSSS5RSSU 5n[R "Xx4SS 9$) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 r"rNr?rrbicubicF)sizemode align_cornersdim)shaperweight unsqueezer6jit is_tracingrrrreshapepermuter functional interpolateviewcat) r^rrrrrrclass_pos_embedpatch_pos_embedr new_height new_widthsqrt_num_positionss r<interpolate_pos_encoding4BridgeTowerVisionEmbeddings.interpolate_pos_encodingsi!&&q)A- !44;;EEaH*003a7 yy##%%+*F6?**4+<+<= =,QU3,QU3r". __, &}c'9:)11!5G]`a)11!Q1=--33 ( 4 *11!Q1=BB1b#Nyy/;CCr; pixel_valuesc ^URup4pVU(dJXPR:wdX`R:wa,[SUSUSURSURS3 5eURRR nURUR US95nURS5RSS5nURRUSS5n [R"X/SS 9n U(aXRXU5-n U $XRUR5-n U $) NzInput image size (*z) doesn't match model (z).rfrr"rr)rr ValueErrorrrrfrjflatten transposerrr6rrrr) r^rr batch_sizerrr target_dtype patch_embeds class_embedsrs r<rs#BridgeTowerVisionEmbeddings.forwards$'3'9'9$ v'V-F%SbSbJb$VHAeW4KDOOK\\]^b^m^m]nnpq ++2288 ++LOO,O,OP #++A.88A> ++22:q"E YY ;C ##&C&CJX]&^^J$&=&=d>O>O&PPJr;) rr_rrrrrrrF)r1r2r3r4r%rQr6ruintrr7rsr:rvrws@r<rrsjq6q,'D5<<'D'DUX'D]b]i]i'DRE$5$5Z_ZfZfr;rc^\rSrSrU4SjrS S\R S\4SjjrS S\R S\4Sjjr S\R 4Sjr S r U=r $) BridgeTowerVisionTransformeric 6>[TU]5 [U5Ul[R "UR URS9Ul[U5Ul [R "UR URS9Ul URUl UR(dg[R"[UR5Vs/sH,n[R "UR URS9PM. sn5Ulggs snfNrJ)rPrQrrrrUrSrVln_prery transformerln_postshare_layernormr~rr| ln_separaters r<rQ%BridgeTowerVisionTransformer.__init__s 5f=ll6#5#56;P;PQ 1&9||F$6$6Fc>[TU]5 URUlURUlURS;aURS:Xa0[R "[ R"S55UlO?URS:Xa/[R "[ R"S55Ul [R"URURS9Ul g[SURS35e) N)add scaled_addrrg?rrrJlink_tower_type  is not implemented) rPrQlink_tower_typerSrrr6tensor scaled_factorbetarUrVNotImplementedErrorr]s r<rQBridgeTowerLinkTower.__init__?s %55!--  ! !%I I%%5%'\\%,,s2C%D"''=8LLc):; \\$*:*:@U@UVDN%(89O9O8PPc&de er;cZURS:XaURX-5$URS:Xa URXR-U-5$URS:Xa0URUSUR- -X R--5$[ SURS35e)Nrrrr"rr)rrUrrr)r^r.cross_modal_hidden_statesrcs r<rsBridgeTowerLinkTower.forwardLs   5 (>>-"KL L  ! !\ 1>>-2D2D"DG`"`a a  ! !] 2>>-1tyy="AD]`i`iDi"ij j%(89M9M8NNa&bc cr;)rUrrSrrr1r2r3r4rQrsr:rvrws@r<rr>s fddr;rcz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $)BridgeTowerSelfOutputiXc(>[TU]5 [R"URUR5Ul[R "URURS9Ul[R"UR5Ul gr) rPrQrrYrSdenserUrVDropouthidden_dropout_probdropoutr]s r<rQBridgeTowerSelfOutput.__init__Ys` YYv1163E3EF f&8&8f>S>STzz&"<"<= r;r. input_tensorrcpURU5nURU5nURX-5nU$rorrrUr^r.rs r<rsBridgeTowerSelfOutput.forward_5 =1  ]3 }'CD r;rUrrrrws@r<rrX6> U\\RWR^R^r;rcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)BridgeTowerIntermediateigc>[TU]5 [R"URUR 5Ul[UR[5(a[URUl gURUl gro) rPrQrrYrSintermediate_sizer isinstance hidden_actstrr intermediate_act_fnr]s r<rQ BridgeTowerIntermediate.__init__hs` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $r;r.rcJURU5nURU5nU$rorrr^r.s r<rsBridgeTowerIntermediate.forwardps& =1 00? r;rrrws@r<rrgs(9U\\ellr;rcz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $)BridgeTowerOutputiwc(>[TU]5 [R"URUR 5Ul[R"UR URS9Ul[R"UR5Ul gr) rPrQrrYrrSrrUrVrrrr]s r<rQBridgeTowerOutput.__init__xs` YYv779K9KL f&8&8f>S>STzz&"<"<= r;r.rrcpURU5nURU5nURX-5nU$ror r s r<rsBridgeTowerOutput.forward~r r;rrrws@r<rrwrr;rcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)BridgeTowerPooleric>[TU]5 [R"URUR5Ul[R "5Ulgro)rPrQrrYrSrTanh activationr]s r<rQBridgeTowerPooler.__init__s9 YYv1163E3EF '')r;r.rc\USS2S4nURU5nURU5nU$)Nr)rr')r^r.first_token_tensor pooled_outputs r<rsBridgeTowerPooler.forwards6+1a40 #56  6 r;)r'rrrws@r<r$r$s($ U\\ellr;r$modulequerykeyvaluercscalingrkwargsc UcURS5S-n[R"XRSS55U-nUbX-n[R R USS9n[R RXURS9n[R"X5n U RSS5R5n X4$)Nrrrr)ptrainingr") rr6matmulrrrsoftmaxrr6 contiguous) r-r.r/r0rcr1rr2 attn_weights attn_outputs r<eager_attention_forwardr<s**R.D(<<}}Q':;gEL!#4 ==((2(>L==((6??([L,,|3K''1-88:K  $$r;c ^\rSrSrS U4SjjrS S\R S\RS-S\S-S\ \ S\ \R 4 S jjr S r U=r$) BridgeTowerSelfAttentioniNcN>[TU]5 URUR-S:wa7[ US5(d&[ SURSURS35eXlURUl[URUR- 5UlURUR-Ul URS-Ul [R"URUR5Ul [R"URUR5Ul[R"URUR5Ul[R "UR"5UlUR&UlX lX0lgNrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()r4)rPrQrSnum_attention_headshasattrrr_rattention_head_size all_head_sizer1rrYr.r/r0rattention_probs_dropout_probr is_decoder is_causal layer_idxr^r_rIrJr`s r<rQ!BridgeTowerSelfAttention.__init__sM     : : :a ?PVXhHiHi#F$6$6#78 445Q8  #)#=#= #&v'9'9F??*9*N*N'&=%C%CI\`\j\j%k "I(?(M(M KK , ,.E) %8      % $}}C$,,..LL %  % ! "));;;;FFH ((r;) rFrEr_rrIrHr/rJrCr.r1r0FN)NNr1r2r3r4rQr6rur7r rrr9rsr:rvrws@r<r>r>sl#648(, ')||')))D0') ') +, ') u||  ')')r;r>c^\rSrSrS U4SjjrS S\R S\RS-S\RS-S\S-S\ \ S \ \R 4 S jjr S r U=r$)BridgeTowerCrossAttentioniNc,>[TU]5 URUR-S:wa7[ US5(d&[ SURSURS35eXlURUl[URUR- 5UlURUR-Ul URS-Ul [R"URUR5Ul [R"URUR5Ul[R"URUR5Ul[R "UR"5UlX lX0lgr@)rPrQrSrCrDrr_rrErFr1rrYr.r/r0rrGrrIrJrKs r<rQ"BridgeTowerCrossAttention.__init__s@     : : :a ?PVXhHiHi#F$6$6#78 445Q8  #)#=#= #&v'9'9FB**4>>:(?(M(M KK , ,.E) %8      % $}}C$,,..LL %  % ! "));;;;FFH ((r;) rFrEr_rrIr/rJrCr.r1r0r]NNN)r1r2r3r4rQr6rur7rrrr9rsr:rvrws@r<r`r`s#4;?376: 1)||1) %00471)))D0 1) -t3 1) +, 1) u|| 1)1)r;r`c^\rSrSrS U4SjjrSS\R S\RS-S\RS-S\RS-S\S-S \ \ S \ \R 4S jjr S r U=r$)BridgeTowerAttentioniDNc>[TU]5 X@lU(a[O[nU"XUS9Ul[ U5Ulg)NrIrJ)rPrQis_cross_attentionr`r>r^routput)r^r_rIrJrqattention_classr`s r<rQBridgeTowerAttention.__init__Es9 "47I3Og#F9U +F3 r;r.rcrcencoder_attention_maskrMr2rc UR(dUOUnUR"U4UUUS.UD6upxURXq5nXx4$)N)rcrcrM)rqr^rr) r^r.rcrcrurMr2attention_outputr:s r<rsBridgeTowerAttention.forwardLs\04/F/FLb)- * "7)+ *  * & ;;'7G--r;)rqrrr^)FNFNNNNr^rws@r<rnrnDs448:>;?(, .||.))D0. %0047 . !& 1 1D 8 .  .+,. u|| ..r;rncP^\rSrSrSU4SjjrSS\\4SjjrSrSr U=r $) BridgeTowerBertCrossLayeriac>[TU]5 URUlSUl[ USUS9UlUR UlURUl[ USUSS9Ul[U5Ul [U5Ul g)Nr"TrpFrIrJrq) rPrQchunk_size_feed_forward seq_len_dimrnrlrHadd_cross_attentioncrossattentionr intermediaterrrr^r_rJr`s r<rQ"BridgeTowerBertCrossLayer.__init__bs '-'E'E$-fPYZ ++#)#=#= 2 #   4F;'/ r;r2c UR"U4USS.UD6upxUn UR"U 4UUUUS.UD6upU n [URURUR U 5n U UU 4$)N)rcrM)rcrcrurM)rlrrfeed_forward_chunkr~r) r^r.rcrcrurMr2self_attention_outputself_attn_weightsrwcross_attention_outputcross_attn_weights layer_outputs r<rs!BridgeTowerBertCrossLayer.forwardrs48>> 4 ) 4  4 0 1595H5H 6 )"7#9+ 6  6 220  # #T%A%A4CSCSUe      r;cJURU5nURX!5nU$rorrrr^rwintermediate_outputrs r<r,BridgeTowerBertCrossLayer.feed_forward_chunk)"//0@A{{#6I r;rrlr~rrrHrrrrorl) r1r2r3r4rQrrrsrr:rvrws@r<r{r{as30(# " +," Hr;r{c^\rSrSrSU4SjjrSS\R S\RS-S\RS-S\RS-S\S-S \ \ S \R 4S jjr S r S r U=r$)BridgeTowerTextLayeriNc>[TU]5 URUlSUl[ XR US9UlUR UlURUlUR(a0UR (d[US35e[ USUSS9Ul [U5Ul [U5Ul g)Nr"rpz> should be used as a decoder model if cross attention is addedFTr})rPrQr~rrnrHrlrrrrrrrrrs r<rQBridgeTowerTextLayer.__init__s '-'E'E$-f@Q@Q]fg ++#)#=#=  # #?? D6)g!hii"6##' #D  4F;'/ r;r.rcrcrurMr2rc 2UR"UU4SU0UD6upxUn UR(a?Ub<[US5(d[SUS35eUR"USUU4SU0UD6upU n [ UR URURU 5n U $)NrMrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`) rlrHrDrrrrr~r) r^r.rcrcrurMr2rrrwrrs r<rsBridgeTowerTextLayer.forwards$(>>  $ ,$  $   1 ??4@4!122 =dVDDD )-(;(;%%& ) !0 )  ) % " 6 0  # #T%A%A4CSCSUe r;cJURU5nURX!5nU$rorrs r<r'BridgeTowerTextLayer.feed_forward_chunkrr;rrory)r1r2r3r4rQr6rur7r rrrsrr:rvrws@r<rrs0.48:>;?(, %||%))D0% %0047 % !& 1 1D 8 %  %+,% %Nr;rc^\rSrSrU4SjrSS\R S\RS-S\RS-S\RS-S\S-S \ S-S \ \ S \ 4S jjr S rU=r$)BridgeTowerTextEncoderic >[TU]5 Xl[R"[ UR 5Vs/sH n[XS9PM sn5Ulgs snf)NrJ) rPrQr_rr~rr|rrr)r^r_ir`s r<rQBridgeTowerTextEncoder.__init__sK  ]]@EfF^F^@_ `@_1 !& 6@_ `  `sANr.rcrcrurM use_cacher2rc nURHnU"UUU4UUS.UD6nM [UU(aUS9$SS9$)N)rurM)last_hidden_staterM)rrr) r^r.rcrcrurMrr2 layer_modules r<rsBridgeTowerTextEncoder.forwards`!JJL(%(> /   M'9+/8O  >B  r;)r_rr)NNNNN)r1r2r3r4rQr6rur7r rkrrrrsr:rvrws@r<rrs 48:>;?(,!% || ))D0  %0047  !& 1 1D 8    $; +,  3  r;rc ^\rSrSrSrU4SjrSS\RS-S\RS-S\RS-S\RS-S \ S \R4 S jjr \ S 5r \ SS j5rSrU=r$)BridgeTowerTextEmbeddingsizGConstruct the embeddings from word, position and token_type embeddings.c>>[TU]5 [R"URUR UR S9Ul[R"URUR 5Ul [R"UR URS9Ul [R"UR5UlURS[ R""UR$5R'S5SS9 URS[ R("UR*R-5[ R.S9SS9 UR Ul[R"UR$UR UR0S9Ulg) N) padding_idxrJrrFrtoken_type_idsr)rPrQrr vocab_sizerS pad_token_idword_embeddingstype_vocab_sizetoken_type_embeddingsrUrVrrrrr6rmax_position_embeddingsrzerosrrlongrposition_embeddingsr]s r<rQ"BridgeTowerTextEmbeddings.__init__s4 !||F,=,=v?Q?Q_e_r_rs%'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<=   ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  "..#%<<  * *F,>,>DL\L\$  r;N input_idsrr inputs_embedspast_key_values_lengthrcUc;UbURXRU5nOURX@R5nUbUR5nOUR5SSnUupxUc[ US5(aQUR R URSS5n [R"U SUS9n U R Xx5nO8[R"U[RURRS9nUcURU5nURU5n XJ-n UR!U5n X-n UR#U 5n UR%U 5n U $)Nrrrr")rindexre)"create_position_ids_from_input_idsr&create_position_ids_from_inputs_embedsrrDrrrr6gatherrrrrgrrrrUr) r^rrrrrrUr seq_lengthbuffered_token_type_idsrrrs r<rs!BridgeTowerTextEmbeddings.forwardsb  $#FF//1G   $JJ=ZjZjk  #..*K',,.s3K!,  !t-..*.*=*=*D*D\EWEWXYEZ\^*_'*/,,7NTU]i*j'!8!?!? !W!&[ SWSdSdSkSk!l   00;M $ : :> J": "66|D5 ^^J/ \\*- r;cUR5SSnUSn[R"US-X1-S-[RURS9nUR S5R U5$)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nrr"rer)rr6rrrgrr)rrrUsequence_lengthrs r<r@BridgeTowerTextEmbeddings.create_position_ids_from_inputs_embedsIsn$((*3B/ %a.|| !O_:Q>ejjYfYmYm %%a(// <rFrM)r.r/cross_attentionsr-cURRn[U[5(GaURRS-SURR -S--nURRS-nSURR-S-nUR RHn[R"URRXB-S9 [R"URR5 [R"URRRX2-S9 [R"UR R"RXR-S9 [R"UR R$RX2-S9 M [R"UR&R(XB-S9 [R"UR&R*RXB-S9 GO[U[,R.[,R0[,R245(a%[R"URSSU-S9 GO[U[,R45(aB[R"UR65 [R8"UR5 GO?[U[:5(a6[R<"UR>URR@5 O[U[B5(aO[RD"URF[HRJ"URL5ROS55 O[U[P5(a{[RD"URF[HRJ"URFRRS5ROS55 [R"URT5 [U[,R.[V45(a/UR6b![R"UR65 ggg) Nr4r)stdrOg?)meanrrr),r_initializer_factorrrrSr|rrinitnormal_rTin_proj_weightzeros_ in_proj_biasout_projrrZrLrOrrrrrYrrrUrones_!BridgeTowerForContrastiveLearning constant_ logit_scalelogit_scale_init_valuercopy_rr6rrrrrrBridgeTowerMLMHead)r^r-rproj_stdattn_stdfc_stdrs r< _init_weights(BridgeTowerPreTrainedModel._init_weightszskk,, f: ; ; //51t{{?\?\;\ae:efH{{..4H$++111d:F++55 UZZ66HNK EJJ334 UZZ0077X^L UYY^^22 E UYY--44(.I 6 LL**:: O LL**==DD(. Y BIIr|| D E E LLSdSj A  - - KK $ JJv}} %  A B B NN6--t{{/Q/Q R  ; < < JJv**ELL9M9M,N,U,UV],^ _  9 : : JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h KK-- . fryy*<= > >6;;CZ KK $D[ >r;r0N)r1r2r3r4r#r8base_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_modules_skip_keys_device_placementrr>r`_can_record_outputsr6no_gradrModulerr:r0r;r<rrlsb %(&+#35ST"3-.5  ]]_%BII%%r;rcR^\rSrSr%\\S'SrU4Sjr\S5r SSjr Sr U=r $) BridgeTowerVisionModelir_)rcd>[TU]U5 [U5UlUR 5 gro)rPrQrvisual post_initr]s r<rQBridgeTowerVisionModel.__init__s&  26:  r;cjURRRRR$ro)rrrrrfr^s r<rfBridgeTowerVisionModel.dtypes${{%%55<<BBBr;c XURURUR5X#5$ro)rtyperf)r^r image_maskrr2s r<rsBridgeTowerVisionModel.forwards {{5::djj1:XXr;)r)NF) r1r2r3r4r%r8rrQpropertyrfrsr:rvrws@r<rrs6 ##! CCYYr;ra0 The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in *Attention is all you need*_ by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. .. _*Attention is all you need*: https://huggingface.co/papers/1706.03762 c^\rSrSr%\\S'SrSU4SjjrSrSr \ \ \ SS\ RS-S \ RS-S \ RS-S \ RS-S \ RS-S \ RS-S\ RS-S\S-S\S-S\\S\4Sjj555rSrSrU=r$)BridgeTowerTextModelir_)rc>[TU]U5 XlSUl[ U5Ul[ U5UlU(a [U5OSUl UR5 g)z^ add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer FN) rPrQr_gradient_checkpointingrrrencoderr$poolerr)r^r_add_pooling_layerr`s r<rQBridgeTowerTextModel.__init__sT   &+#3F;-f5 3D'/$  r;c.URR$rorrrs r<get_input_embeddings)BridgeTowerTextModel.get_input_embeddingss...r;c$XRlgrorr^r0s r<set_input_embeddings)BridgeTowerTextModel.set_input_embeddingss */'r;NrrcrrrrcrurMrr2rc USLUSL- (a [S5eURR(dSn U (a1Uc.[[ URS9[ URS95nUbUR 5OSn UR UUUUU S9n URUUU UUS9up'UR"U 4UUUUU US.U D6n U SnURbURU5OSn[UUU RS9$) Nz:You must specify exactly one of input_ids or inputs_embedsF)r_r)rrrrr)rcruembedding_outputrcrM)rcrcrurMrr)rr-rM) rr_rHrr get_seq_lengthr_create_attention_masksrrrrM)r^rrcrrrrcrurMrr2rrencoder_outputssequence_outputr+s r<rsBridgeTowerTextModel.forwards8&  -t"; <YZ Z{{%%I 01,dkk2RT`hlhshsTtuOETE`!?!?!Afg??%)'#9 + 261M1M)#9-"7+ 2N2 .,,   )"7#9+%    *!,8< 8O O4UY ;-'+;;  r;cURR(a[URUUUS9nO[URUUS9nUb[URUUUS9nX4$)N)r_rrcrM)r_rrc)r_rrcrc)r_rHrr)r^rcrurrcrMs r<r,BridgeTowerTextModel._create_attention_maskssr ;; ! !/{{.- / N7{{.-N " -%>{{.5&; & "55r;)r_rrrr)T) NNNNNNNNN)r1r2r3r4r$r8rrQr r r r!rr6rur rkrrrrsrr:rvrws@r<rrs% "! "/0  *..2.2,0-1596:(,!%9 <<$&9  t+9  t+ 9 llT) 9 ||d* 9  %||d29 !& t 39 9 $;9 +,9  69  9 x66r;rzv The bare BridgeTower Model transformer outputting BridgeTowerModelOutput object without any specific head on c^\rSrSrU4SjrSrSrS\RS\ S\R4Sjr S\RS\ S\R4S jr \ \ SS \RS -S \RS -S \RS -S\RS -S\RS -S\RS -S\RS -S\ S -S\RS -S\S\\S\\R\-4Sjj55rSrSrU=r$)BridgeTowerModeli8c  >[TU]U5 XlURnURnUR (aa[ R"URUR5Ul [ R"URUR5Ul O[ R"[UR5Vs/sH.n[ R"URUR5PM0 sn5Ul [ R"[UR5Vs/sH.n[ R"URUR5PM0 sn5Ul [ R"SUR5Ul[!U5Ul[%U5UlUR((dUR*(aUR"R,R.HnUR"R,R0R2R4UR2lUR"R,R0R6R4UR6lM [ R"[UR5Vs/sHn[9U5PM sn5Ul[ R"[UR5Vs/sHn[9U5PM sn5Ul[?U5Ul [?U5Ul![ RD"URURFS9Ul$[ RD"URURFS9Ul%URL(a![OU5Ul([OU5Ul)O[ R"[URS- 5Vs/sHn[OU5PM sn5Ul([ R"[URS- 5Vs/sHn[OU5PM sn5Ul)URU5 gs snfs snfs snfs snfs snfs snf)NrrJr")+rPrQr_ vision_config text_config$share_cross_modal_transformer_layersrrYrScross_modal_text_transformcross_modal_image_transformr~rr|rrr vision_modelr text_modelr"init_layernorm_from_vision_encoderrcross_modal_ln_separaterrdatarr{cross_modal_image_layerscross_modal_text_layersr$cross_modal_image_poolercross_modal_text_poolerrUrVcross_modal_text_layernormcross_modal_image_layernormshare_link_tower_layersrcross_modal_text_link_towercross_modal_image_link_towerr)r^r_rrrrr`s r<rQBridgeTowerModel.__init__>sU   ,, ((  6 6.0ii 8O8OQWQcQc.dD +/1yy9R9RTZTfTf/gD ,.0mmQVW]WoWoQpqQpA;22F4F4FGQpq/D +02}}SXY_YqYqSrsSra=44f6H6HISrs0D ,&(\\!V5G5G%H"2=A.{;,,1Z1Z''..FF!%!2!2!9!9!A!A!H!H!M!M #0077??DDII G)+ =B6C[C[=\ ]=\ &{ 3=\ ]) %(*}}=B6C[C[=\ ]=\ &{ 3=\ ]( $ ):&(A%'8'@$+-,,v7I7IvOdOd*e'+-<<8J8JPVPePe+f(  ) )/CF/KD ,0DV0LD -/1}}7"S U[ RURS95R9U5nUU-nURAU5n[ R "URS5URS54[ RURS9nURRXUR55RUR5nURBS"UUUUS 9nUSnURDS"UUUUS 9nUSnUR UU45 UR USUS45 Sn [GU[URRR55GHn!URRRU!"UU5nUR R"R,R.U!"U5R'UR R(5nUR=UR R"R1U5U S-5U-nURHU n"URJU n#UR3UU S-5n$U""U$U-UU5n%U#"UUU5n&URBU S-"U%U&UUS 9nUSnURDU S-"U&U%UUS 9nUSnU S- n U R U5 U R U5 UR UU45 UR USUS45 GM UUn(n'URMU'U(5n)[OU'U(U)[QU 5[QU 5[QU54[QU5S 9$) a image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. image_token_type_idx (`int`, *optional*): - The token type ids for images. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels are currently not supported. Examples: ```python >>> from transformers import BridgeTowerProcessor, BridgeTowerModel >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> # prepare image and text >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> text = "hello world" >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base") >>> model = BridgeTowerModel.from_pretrained("BridgeTower/bridgetower-base") >>> inputs = processor(image, text, return_tensors="pt") >>> outputs = model(**inputs) >>> outputs.keys() odict_keys(['text_features', 'image_features', 'pooler_output']) ```NzYBridgeTowerModel does not use `inputs_embeds`. Make sure to pass in `input_ids` instead.r")rrerrrrr")rcru)r+r,r-r.r/))rrr!rrr6onesrrgget_extended_attention_maskrjlenrrrr_r|r rrrrfrrrrr5rr expand_asr)r8fullr*r&r%rr,r-get_cls_featuresr)r9)*r^rrcrrr:rrCr;r<rr2all_hidden_states_textall_hidden_states_imageall_hidden_states_crossall_self_attentionsrUrBextend_text_masks split_indexrrrimage_embeds_with_lncross_modal_texttext_token_type_embeddingsimage_token_type_embeddingscross_modal_imageextend_image_maskslayer_outputs_textcross_text_featureslayer_outputs_imagecross_image_featureslink_layer_indexrtext_link_towerimage_link_towertransformed_text_embedscross_text_features_cross_image_features_r+r, cls_featuress* r<rsBridgeTowerModel.forwardsf\"$"$"$   $):%k  48qnn& oo0090E %%k2  !"ZZ 5::iN^N^_N OOGGXcdgg     $//117784;;;X;XX[\\ __,,22?K " ) )+ 6A  ,,33??!!$"3"3"9"9:Um@L (//1a8L&&|4&&--99CCL[QE .L # * *< 8R $0077DD\EVEVW[WhWhWnWnEop 55kQ5O%)%?%? KKI4D4D E& )$ % # ::;KNh;hi#::;O[\:]&*&@&@ JJt1IL\L\ ]' )( ) $ 46QQ <<=QRZZ  # #A &(9(>(>q(A B**## "__HHUdUdUfgjj    "99!<  ,#5   13";;A>  -#4   315&&(;=Q'RS""$6q$9;Nq;Q#RS{C(?(?(E(E$FGA//1177:;HYZK,,33??II!L\Z__!!''L++D,=,=,D,D,Q,QR^,_aqtuauv-. ! #>>?OPO#@@AQR '+&@&@N^abNb&c ##2'*DD#!$ %55IK_as$t !"&!=!=>NQR>R!S$%0'9 "  #5Q"7 "&"?"?@PST@T"U%$1'8 #  $7q#9  !  " ) )+ 6 # * *< 8 # * *,?AU+V W  & &(:1(=?RST?U'V W[H`)<=Q~ ,,]NK %')&,--.-. 01  r;crURU5nURU5n[R"X4/SS9$)Nrr)r(r'r6r)r^r+r,cls_features_textcls_features_images r<rD!BridgeTowerModel.get_cls_featuresSs9 88G!::>Jyy+@bIIr;)r_r*r%r-r'rr)r&r,r(rr!rr ) NNNNNNNNNF)r1r2r3r4rQr r r6rurr5r8rrrr7rkrrr9r)rsrDr:rvrws@r<rr8s 6p64I5<<ICITYT`T`I JELLJSJUZUaUaJ .2372615.22615+/*.).I ##d*I ))D0I ((4/ I ''$. I $$t+ I ((4/I ''$.I "DjI   4'I #'I +,I  u|| 5 5I I VJJr;rc.^\rSrSrU4SjrSrSrU=r$)"BridgeTowerPredictionHeadTransformiZcp>[TU]5 [R"URUR5Ul[ UR[5(a[URUl OURUl [R"URURS9Ul gr) rPrQrrYrSrrrrr transform_act_fnrUrVr]s r<rQ+BridgeTowerPredictionHeadTransform.__init__[s~ YYv1163E3EF f'' - -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STr;clURU5nURU5nURU5nU$ro)rrdrUrs r<rs*BridgeTowerPredictionHeadTransform.forwardds4 =1 --m< }5 r;)rUrrdrrws@r<rbrbZsUr;rbc2^\rSrSrSU4SjjrSrSrU=r$)rikcn>[TU]5 Xl[U5Ul[ R "URURRSS9Ul [ R"[R"URR55UlUbX Rlgg)NF)r)rPrQr_rb transformrrYrSrrdecoderrr6rrr)r^r_rr`s r<rQBridgeTowerMLMHead.__init__ls  ;FCyy!3!3V5G5G5R5RY^_ LLV-?-?-J-J!KL  "(LL  r;cdURU5nURU5UR-nU$ro)rjrkr)r^x mlm_scores r<rsBridgeTowerMLMHead.forwardus-NN1% LL+dii7 r;)rr_rkrjrorrws@r<rrks)r;rc.^\rSrSrU4SjrSrSrU=r$)BridgeTowerITMHeadi{cZ>[TU]5 [R"US5UlgNrrPrQrrYfc)r^rSr`s r<rQBridgeTowerITMHead.__init__|s  ))K+r;c(URU5nU$rorv)r^rn itm_scores r<rsBridgeTowerITMHead.forwardsGGAJ r;ryrrws@r<rrrr{s,r;rrz\ BridgeTower Model with a language modeling head on top as done during pretraining. cl^\rSrSrSS0rU4SjrSrSr\\ SS\ RS-S \ RS-S \ RS-S \ RS-S \ RS-S \ RS-S\ RS-S\ RS-S\ \S\4Sjj55rSrU=r$)BridgeTowerForMaskedLMizmlm_score.decoder.weightz8bridgetower.text_model.embeddings.word_embeddings.weightc>[TU]U5 [U5Ul[ U5UlUR 5 gro)rPrQrrrrorr]s r<rQBridgeTowerForMaskedLM.__init__s5  +F3+F3 r;c.URR$rororkrs r<get_output_embeddings,BridgeTowerForMaskedLM.get_output_embeddingss~~%%%r;c$XRlgror)r^new_embeddingss r<set_output_embeddings,BridgeTowerForMaskedLM.set_output_embeddingss !/r;Nrrcrrr:rrCr<r2rc UR"SUUUUUUUS.U D6n URU R5n Sn Ubk[5n UR U R 5nU "U R SURRR5UR S55n [U U U RU RS9$)a image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import BridgeTowerProcessor, BridgeTowerForMaskedLM >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> url = "http://images.cocodataset.org/val2017/000000360943.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())).convert("RGB") >>> text = "a looking out of the window" >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") >>> model = BridgeTowerForMaskedLM.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> results = processor.decode(outputs.logits.argmax(dim=-1).squeeze(0).tolist()) >>> print(results) .a cat looking out of the window. ```rrcrrr:rrCNrr@rAr.r/r0) rror+rrjrgrr_rrrr.r/)r^rrcrrr:rrCr<r2outputs mlm_logitsmasked_lm_lossloss_fcts r<rsBridgeTowerForMaskedLM.forwardsd""  ))%!'%    ^^G$9$9:   ')HYYz001F%joob$++:Q:Q:\:\&]_e_j_jkm_noN!//))   r;)rroNNNNNNNN)r1r2r3r4_tied_weights_keysrQrrrrr6rr7rrrrsr:rvrws@r<r}r}s" 56pq&0.2372615.22615*.H ##d*H ))D0H ((4/ H ''$. H $$t+ H ((4/H ''$.H   4'H +,H  H H r;r}z BridgeTower Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for image-to-text matching. cX^\rSrSrU4Sjr\\SS\RS-S\RS-S\RS-S\RS-S\RS-S \RS-S \RS-S \RS-S \ \ S \ 4Sjj55r SrU=r$)#BridgeTowerForImageAndTextRetrievalic>[TU]U5 [U5Ul[ UR S-5UlUR5 grt)rPrQrrrrrSrzrr]s r<rQ,BridgeTowerForImageAndTextRetrieval.__init__s@  +F3+F,>,>,BC r;Nrrcrrr:rrCr<r2rc UR"SUUUUUUUS.U D6n U Rn URU 5n Sn Ub-[5nUR U R 5nU"X5n [ U U U RU RS9$)a image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size, 1)`, *optional*): Labels for computing the image-text matching loss. 0 means the pairs don't match and 1 means they match. The pairs with 0 will be skipped for calculation. Examples: ```python >>> from transformers import BridgeTowerProcessor, BridgeTowerForImageAndTextRetrieval >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> texts = ["An image of two cats chilling on a couch", "A football player scoring a goal"] >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") >>> model = BridgeTowerForImageAndTextRetrieval.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") >>> # forward pass >>> scores = dict() >>> for text in texts: ... # prepare inputs ... encoding = processor(image, text, return_tensors="pt") ... outputs = model(**encoding) ... scores[text] = outputs.logits[0, 1].item() ```rNrr0) rr-rzrrjrgrr.r/)r^rrcrrr:rrCr<r2rr-rAitm_lossrs r<rs+BridgeTowerForImageAndTextRetrieval.forwards\""  ))%!'%     --  .  ')HYYv}}-F/H'!//))   r;)rrzr)r1r2r3r4rQrrr6rr7rrrrsr:rvrws@r<rrs .2372615.22615*.G ##d*G ))D0G ((4/ G ''$. G $$t+ G ((4/G ''$.G   4'G +,G  "G G r;rc.^\rSrSrU4SjrSrSrU=r$)BridgeTowerContrastiveHeadiFcX>[TU]5 [R"X5Ulgroru)r^rS embed_sizer`s r<rQ#BridgeTowerContrastiveHead.__init__Gs ))K4r;c(URU5nU$rory)r^rns r<rs"BridgeTowerContrastiveHead.forwardKs GGAJr;ryrrws@r<rrFs5r;rzl BridgeTower Model with a image-text contrastive head on top computing image-text contrastive loss. cD^\rSrSrU4Sjr\\SS\RS-S\RS-S\RS-S\RS-S\RS-S \RS-S \RS-S \ S-S \ \ S \ 4Sjj55rSrU=r$)riPc>[TU]U5 [U5Ul[ UR UR 5Ul[ UR UR 5Ul[ UR S-UR 5Ul [R"[R"URR55UlUR#5 grt)rPrQrrrrScontrastive_hidden_size itc_text_headitc_image_headitc_cross_modal_headrrr6rr_rrrr]s r<rQ*BridgeTowerForContrastiveLearning.__init__Vs  +F378J8JFLjLjk89K9KVMkMkl$>v?Q?QTU?UW]WuWu$v!<< T[[5W5W(XY r;Nrrcrrr:rrC return_lossr2rc U RSS5 UR"SUUUUUUUS.U D6n U Rn U RupnU SnU SnURRR R U5nURR[R"SS[RURRRRS95RU5nURRU5U-n[R R#UR%USS2S SS245SS S 9n[R R#UR'USS2S SS245SS S 9R)URS 9n[R R#UR+U 5SS S 9R)URS 9n[R,"XU/S S9nUR.R15R)URS 9n[R2"XR555U-n[R2"UUR555U-n[R2"UUR555U-nSnU(a[R6"[9U5URS 9n[R R;UU5n[R R;UU5n[R R;UU5nUU-U-S- n[=UUUUUU RU R>S9$)a image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. Examples: ```python >>> from transformers import BridgeTowerProcessor, BridgeTowerForContrastiveLearning >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> import torch >>> image_urls = [ ... "https://farm4.staticflickr.com/3395/3428278415_81c3e27f15_z.jpg", ... "http://images.cocodataset.org/val2017/000000039769.jpg", ... ] >>> texts = ["two dogs in a car", "two cats sleeping on a couch"] >>> with httpx.stream("GET", urls[0]) as response: ... image1 = Image.open(BytesIO(response.read())) >>> with httpx.stream("GET", urls[1]) as response: ... image2 = Image.open(BytesIO(response.read())) >>> images = [image1, image2] >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-large-itm-mlm-itc") >>> model = BridgeTowerForContrastiveLearning.from_pretrained("BridgeTower/bridgetower-large-itm-mlm-itc") >>> inputs = processor(images, texts, padding=True, return_tensors="pt") >>> loss = model(**inputs, return_loss=True).loss >>> inputs = processor(images, texts[::-1], padding=True, return_tensors="pt") >>> loss_swapped = model(**inputs, return_loss=True).loss >>> print("Loss", round(loss.item(), 4)) Loss 0.0019 >>> print("Loss with swapped images", round(loss_swapped.item(), 4)) Loss with swapped images 2.126 ```output_hidden_statesTrrr>r"reNrr)rr5)rgrg@)r@rArBrCrDr.r/r0) setdefaultrr-r.r rrrr6rCrrrgrBrrr normalizerrrjrrrexpr7trrA cross_entropyr>r/)r^rrcrrr:rrCrr2rr-hidden_states_txthidden_states_imghidden_states_cross_modalrBrKrNrDrArlogits_text_to_imagelogits_text_to_crosslogits_image_to_crossitc_lossr<text_to_image_losstext_to_cross_lossimage_to_cross_losss r<rs)BridgeTowerForContrastiveLearning.forwardcs"v 0$7""  ))%!'%     -- JQJ_J_G.G'+ (, #//<<CCPPQ]^&*&6&6&L&L JJtQejj9I9I9_9_9f9f9m9m n' )( ) $''CCDXY\ww mm--d.@.@QPQSTWAU.V\^bc-d }}..t/B/B>;K L{ Z \\#f+fmmDF!#!SU["\ *-??BUUY\\H+#%%!//))  r;)rrrrrr)r1r2r3r4rQrrr6rr7rkrrr>rsr:rvrws@r<rrPs .2372615.22615#'s ##d*s ))D0s ((4/ s ''$. s $$t+ s ((4/s ''$.s D[s +,s  &s s r;r)rrr}rr)NrO)[r5 collectionsrcollections.abcr dataclassesrr6rtorch.nnrr r activationsr r cache_utilsr r r masking_utilsrrmodeling_layersrmodeling_outputsrrrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrutilsrrrr utils.genericrr utils.output_capturingr!configuration_bridgetowerr#r$r% get_loggerr1logger_TOKENIZER_FOR_DOCr)r>rrFryrrrrrrr$rufloatr<r>r`rnr{rrrrrrrrbrrrr}rrr__all__r0r;r<rs #$! %&6CCJ9G&6KKI5hh   H %'  7[7  7$  7;7  74)299)XRYY6P"))Pf7"2997"td299d4 BII  bii     ,!% II% <<% % << % LL4' % T\ %%'(%:@)ryy@)HI) I)Z.299.:8 8v?5?F RYY Fg8 g8T+%+%+%\Y7Y"  {65{6 {6| YJ1YJ  YJz"    \ 7\  \ ~ T *DT  T n C (BC  C L r;