Z jC S r SSKrSSKJr SSKJr SSKrSSK r SSK J
r
SSK J r
SSKJr SS KJr SS
KJr SS KJrJr SS KJr SS
KJr SSKJ r J r J r J r J r J!r! SSK"J#r# SSK$J%r% SSK&J'r'J(r(J)r) \ RT " \+5 r,S\ RZ S\ RZ 4S jr.S\ RZ S\ RZ 4S jr/S\ RZ S\04S jr1SHS\ RZ S\2S\3S\0S\ RZ 4
S jjr4SIS jr5S r6 " S S \
Rn 5 r8 " S! S"\
Rn 5 r9 " S# S$\
Rn 5 r:\\ " S% S&\ 5 5 5 r; " S' S(\
Rn 5 r< " S) S*\
Rn 5 r= " S+ S,\
Rn 5 r> " S- S.\
Rn 5 r? " S/ S0\
Rn 5 r@ " S1 S2\@5 rA " S3 S4\
Rn 5 rB " S5 S6\5 rC\ " S7 S8\5 5 rD " S9 S:\
Rn 5 rE " S; S<\
Rn 5 rF " S= S>\D5 rG " S? S@\D5 rH " SA SB\
Rn 5 rI " SC SD\D5 rJ\ " SE SF\D5 5 rK/ SGQrLg)JzPyTorch GroupViT model. N) dataclass)Any)nn )initialization)ACT2FN)create_causal_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPooling)PreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tuplelogging torch_int)merge_with_config_defaults)capture_outputs )GroupViTConfigGroupViTTextConfigGroupViTVisionConfiglogitsreturnc [ R R U [ R " [ U 5 U R S95 $ )Ndevice)r
functional
cross_entropytorcharangelenr )r s /root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/groupvit/modeling_groupvit.pycontrastive_lossr& * s/
== & &vu||CKPVP]P]/^ __
similarityc P [ U 5 n[ U R 5 nX- S- $ )Ng @)r& T)r( caption_loss
image_losss r% image_text_contrastive_lossr- / s( #J/L!*,,/J % ,,r' dimc U R U5 nUR USS9S n[ R " U [ R S9R XS5 nXBR
5 -
U- nU$ )NTkeepdimr
memory_format ?)softmaxmaxr"
zeros_likelegacy_contiguous_formatscatter_detach)r r. y_softindexy_hardrets r% hard_softmaxr? 5 sf
^^C
F JJsDJ )! ,E
fE4R4R
S
\
\]`il
mF
==?
"V
+C Jr' tauhardc , [ R R R [ R " SU R
U R S9[ R " SU R
U R S95 nUR U R 5 nX- U- nUR U5 nU( a] UR USS9S n[ R " U [ R S9R X7S5 nXR 5 -
U- n U $ Un U $ )N )r dtyper4 Tr0 r r2 )r"
distributionsgumbelGumbeltensorr rD sampleshaper5 r6 r7 r8 r9 r: )
r r@ rA r. gumbel_distgumbelsr; r< r= r> s
r% gumbel_softmaxrM ? s %%,,33
S fllC
S fllCK .G 3&G
__S
!F
3
-a0!!&8V8VW``admpq}}&/ J Jr' c X- U R S - S- nX: a4 [ [ R " X$- 5 5 nU R S U- nO3[ [ R " X- 5 5 nU R S U- nU R S nU R S nU R XxXe5 n [
R R XU4SUS9n U $ )a
Args:
attentions (`torch.Tensor`): attention map of shape [batch_size, groups, feat_height*feat_width]
height (`int`): height of the output attention map
width (`int`): width of the output attention map
align_corners (`bool`, *optional*): the `align_corner` argument for `nn.functional.interpolate`.
Returns:
`torch.Tensor`: resized attention map of shape [batch_size, groups, height, width]
?r r bilinearsizemode
align_corners)rJ intnproundreshaper r interpolate)
attentionsheightwidthrU scale
feat_width feat_height
batch_sizegroupss r% resize_attention_maprc U s ^z / / 2
2s :E
~%-01
&&q)Z7 "((6>23 %%a(K7
!!!$J
a
F##J PJ **% z + J r' c V / n[ R " 5 SnU Hj nUR SSS5 R 5 nUc UnOX4- n[ UR SSS5 R 5 /UQ76 nUR U5 Ml SSS5 US nU$ ! , ( d f N= f)a
Args:
attentions (`tuple(torch.FloatTensor)`: tuple of attention maps returned by `GroupViTVisionTransformer`
hw_shape (`tuple(int)`): height and width of the output attention map
Returns:
`torch.Tensor`: the attention map of shape [batch_size, groups, height, width]
Nr rO r )r" no_gradpermute
contiguousrc append)r[ hw_shape attn_mapsprev_attn_masks
attn_masks cur_attn_mapfinal_groupings r% get_grouping_from_attentionsrp s s I $J#++Aq!4??AJ &","1">/0G0G1a0P0[0[0]i`hiL \ * %
r]N !
s A3B
B(c 6 ^ \ rS rSrS\4U 4S jjrS rSrU =r$ )GroupViTCrossAttentionLayer configc > [ TU ] 5 [ U5 U l [ R
" UR UR S9U l [ U5 U l
[ R
" UR UR S9U l g Neps) super__init__GroupViTAttentionattnr LayerNorm hidden_sizelayer_norm_epsnorm2 GroupViTMLPmlp norm_postselfrt __class__s r% rz $GroupViTCrossAttentionLayer.__init__ sb
%f- \\&"4"4&:O:OP
v& f&8&8f>S>STr' c UnX0R XS9S - nX0R U R U5 5 - nU R U5 nU$ )N)encoder_hidden_statesr r| r r r )r querykeyxs r% forward#GroupViTCrossAttentionLayer.forward sK
% ;A> >
A ' ' NN1 r' r ) __name__
__module__ __qualname____firstlineno__r rz r __static_attributes__
__classcell__r s @r% rr rr s U 3 U r' rr c @ ^ \ rS rSrS\4U 4S jjrSS jrS rSrU =r $ )GroupViTAssignAttention rt c > [ TU ] 5 UR S- U l [ R
" UR UR 5 U l [ R
" UR UR 5 U l [ R
" UR UR 5 U l [ R
" UR UR 5 U l UR U l
g )N ) ry rz r~ r^ r Linearq_projk_projv_projproj
assign_epsr s r% rz GroupViTAssignAttention.__init__ s
''-
ii 2 2F4F4FG ii 2 2F4F4FG ii 2 2F4F4FG IIf 0 0&2D2DE ++r' c U( a U R ( a
[ USUS9nU$ U( a [ USS9nU$ [ R R USS9nU$ )N)r. rA r. )trainingrM r? r r r5 )r r| rF rA s r% get_attn GroupViTAssignAttention.get_attn sX dmm!$BT:D #Db1 }},,Tr,: r' c ` UnU R U5 nU R U5 nU R U5 nXR SS5 - U R - nU R U5 nU R USSS9nXUR
SSS9U R - - nXS- nU R U5 nXv4$ )Nr re F)rF rA Tr. r1 ) r r r transposer^ r sumr r )r r r valueraw_attnr| soft_attnouts r% r GroupViTAssignAttention.forward s E" kk# E" MM"b 11TZZ?}}X&MM(5uME xxBx5GHliin~ r' )r r r r r^ r )TT)
r r r r r rz r r r r r s @r% r r s , 3 , r' r c < ^ \ rS rSrS\4U 4S jjrS rS rSrU =r $ )GroupViTTokenAssign rt c V > [ TU ] 5 X0l [ R " UR
UR S9U l [ UR [ R R 5 ( a UR OUR UR 4nU Vs/ s H n[ XQR
- 5 PM snu pg[ XXc5 U l [ R " UR
UR S9U l [ R " UR
UR S9U l [% U5 U l [) U5 U l [ R " UR
UR S9U l [/ XR
XqR
5 U l g s snf rv )ry rz num_output_groupr r} r~ r norm_tokens
isinstanceassign_mlp_ratio collectionsabcIterablerV GroupViTMixerMLP mlp_internorm_post_tokensnorm_xrr pre_assign_attnr assign
norm_new_xr mlp_channels) r rt num_group_tokenr r r
tokens_dim channels_dimr s r% rz GroupViTTokenAssign.__init__ sD
0 <<(:(:@U@UV &11;??3K3KLL
# #))6+B+BC
JZ#ZIYAC,>,>(>$?IY#Z
)&:` " V-?-?VEZEZ [ ll6#5#56;P;PQ :6 B -f5 ,,v'9'9v?T?TU '0B0BLRdRd e $[s !F&c J U R U5 nU R U5 nU$ )z
Args:
group_tokens (torch.Tensor): group tokens, [batch_size, num_group_tokens, channels]
Returns:
projected_group_tokens (torch.Tensor): [batch_size, num_output_groups, channels]
)r r )r group_tokensprojected_group_tokenss r% project_group_token'GroupViTTokenAssign.project_group_token s+ "& != !%!6!67M!N %%r' c U R U5 nU R U5 nU R U5 nU R X15 nU R X15 u pEXC-
nX@R U R
U5 5 - nXE4$ )z
Args:
image_tokens (`torch.Tensor`): image tokens, of shape [batch_size, input_length, channels]
group_tokens (`torch.Tensor`): group tokens, [batch_size, num_group_tokens, channels]
)r r r r r r r )r image_tokensr r new_image_tokens attentions r% r GroupViTTokenAssign.forward s '' 5 {{<0 !%!9!9,!G !%!5!56L![ &*kk2H&W#2+.?.?P`@a.bb **r' ) r r r r r r r r r )
r r r r r rz r r r r r s @r% r r s! f 3 f* &+ +r' r c : \ rS rSr% SrSr\R S- \S' Sr \R S- \S' Sr
\R S- \S' Sr \R S- \S' Sr \R S- \S' Sr
\R S- \S ' Sr\\S
' Sr\\S ' S \\ 4S
jrSrg)GroupViTModelOutputi a
loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`):
Contrastive loss for image-text similarity.
logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`):
The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text
similarity scores.
logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`):
The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image
similarity scores.
segmentation_logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels, logits_height, logits_width)`):
Classification scores for each pixel.
The logits returned do not necessarily have the same size as the `pixel_values` passed as inputs. This is
to avoid doing two interpolations and lose some quality when a user needs to resize the logits to the
original image size as post-processing. You should always check your logits shape and resize as needed.
text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`):
The text embeddings obtained by applying the projection layer to the pooled output of
[`GroupViTTextModel`].
image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`):
The image embeddings obtained by applying the projection layer to the pooled output of
[`GroupViTVisionModel`].
text_model_output (`BaseModelOutputWithPooling`):
The output of the [`GroupViTTextModel`].
vision_model_output (`BaseModelOutputWithPooling`):
The output of the [`GroupViTVisionModel`].
Nlosslogits_per_imagelogits_per_textsegmentation_logits text_embeds image_embedstext_model_outputvision_model_outputr c J ^ [ U 4S jT R 5 5 5 $ )Nc 3 n ># U H* nUS ; a TU O[ TU5 R 5 v M, g7f))r r N)getattrto_tuple).0kr s r% /GroupViTModelOutput.to_tuple..3 s<
LLDGRYZ^`aRbRkRkRm m s 25)tuplekeysr s `r% r GroupViTModelOutput.to_tuple2 s#
YY[
r' )r r r r __doc__r r" FloatTensor__annotations__r r r r r r r r r r r r r r' r% r r s > &*D%
d
")15e''$.504OU&&-448**T18,0K""T)0-1L%##d*148186:3:
%*
r' r c ^ \ rS rSrSr S
S\\\ - \\\4 - S\\\\4 - S\S\4U 4S jjjrSS\ R S \ S
\ R 4S jjr S r
U =r$ )GroupViTPatchEmbeddingsi9 z
Image to Patch Embedding.
image_size
patch_size num_channels embed_dimc ` > [ TU ] 5 [ U[ R R
5 ( a UOX4n[ U[ R R
5 ( a UOX"4nUS US - US US - - nXl X l XPl [ R " X4X"S9U l g )Nr r ) kernel_sizestride) ry rz r r r r r r num_patchesr Conv2d
projection)r r r r r r r s r% rz GroupViTPatchEmbeddings.__init__> s #-j+//:R:R#S#SZZdYq
#-j+//:R:R#S#SZZdYq
!!}
1
5*Q-:VW=:XY $$& ))Lgr' pixel_valuesinterpolate_pos_encodingr c > UR u p4pVU( dV XPR S :w d X`R S :w a2 [ SU SU SU R S SU R S S3 5 eU R U5 R S5 R SS5 nU$ )Nr r zInput image size (*z) doesn't match model ().rO )rJ r
ValueErrorr flattenr )r r r ra r r\ r] r s r% r GroupViTPatchEmbeddings.forwardO s 2>2D2D/
&' ++u8J/J (% 9 +,Adooa.@-AE
OOL ) 1 1! 4 > >q! Dr' )r r r r ) r i F)r r r r r rV listr rz r" Tensorboolr r r r s @r% r r 9 s 9<,. h$s)OeCHo5h %S/)h h
h h" ELL D ]b]i]i r' r c ^ \ rS rSrS\4U 4S jjrS\R S\S\S\R 4S jr S
S \R S
\
S\R 4S jjr S r U =r
$ )GroupViTVisionEmbeddingsi[ rt c > [ TU ] 5 [ UR UR UR
UR S9U l U R R n[ R " [ R " SX!R 5 5 U l
[ R " UR 5 U l [ R " UR UR" S9U l UR U l Xl g )N)r r r r r rw )ry rz r r r r r~ patch_embeddingsr r Parameterr" zerosposition_embeddingsDropoutdropoutr} r layernormrt )r rt r r s r% rz !GroupViTVisionEmbeddings.__init__\ s
7 (( (( ,, (( !
++77 #%<< A{L^L^0_#` zz&..1 f&8&8f>S>ST ++ r'
embeddingsr\ r] r c , UR S nU R R S n[ R R 5 ( d XE:X a X#:X a U R $ U R nUR S nX R
- nX0R
- n [
US- 5 n
UR SXU5 nUR SSSS5 n[ R R UX4SSS 9nUR SSSS5 R SSU5 nU$ )
a
This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution
images. This method is also adapted to support torch.jit tracing and no class embeddings.
Adapted from:
- https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and
- https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211
r re rP r r rO bicubicFrR )
rJ r r" jit
is_tracingr r rY rg r r rZ view) r r r\ r] r
num_positionspatch_pos_embedr.
new_height new_widthsqrt_num_positionss r% r 1GroupViTVisionEmbeddings.interpolate_pos_encodingl s !&&q) 0066q9
yy##%%+*F6?++ + 22 r".
__, &}c'9 :)11!5G]`a)11!Q1= --33 ( 4
*11!Q1=BB1b#N r' r r c UR u p4pVU R XS9nU R U5 nUR 5 u p8n U( a XpR XuU5 - nOXpR
- nU R
U5 nU$ )N)r )rJ r r rS r r r )
r r r ra r r\ r] r seq_len_s
r% r GroupViTVisionEmbeddings.forward s 2>2D2D/
&**<*k
^^J/
!+ !2
Q $#&C&CJX]&^^J#&>&>>J\\*-
r' )rt r r r r r r )r r r r r rz r" r rV r r r r r r s @r% r r [ sh 3 $ 5<< $ $ UX $ ]b]i]i $ LELL D ]b]i]i r' r c ^ \ rS rSrS\4U 4S jjr S S\R S- S\R S- S\R S- S\R 4S jjr
S
r U =r $ ) GroupViTTextEmbeddingsi rt c N > [ TU ] 5 UR n[ R " UR
U5 U l [ R " UR U5 U l U R S[ R " UR 5 R S5 SS9 g )N position_idsr re F)
persistent)
ry rz r~ r Embedding
vocab_sizetoken_embeddingmax_position_embeddingsposition_embeddingregister_bufferr" r# expandr rt r r s r% rz GroupViTTextEmbeddings.__init__ s
&& !||F,=,=y I "$,,v/M/My"Y
ELL)G)G H O OPW Xej
r' N input_idsr&
inputs_embedsr c < Ub UR S OUR S nU R R R S nXE: a [ SU SU 35 eUc U R S S 2S U24 nUc U R U5 nU R U5 nX6- nU$ )Nre r r zRSequence length must be less than max_position_embeddings (got `sequence length`: z and max_position_embeddings: )rJ r- weightr r& r+ )r r2 r& r3
seq_lengthmax_position_embeddingr r s r% r GroupViTTextEmbeddings.forward s -6,AY__R(}GZGZ[]G^
!%!8!8!?!?!E!Ea!H . d , <=S [ TU ] 5 X l X@l US: a; [ R
" [ R " SXAR 5 5 U l OS U l [ R " [ U5 Vs/ s H n[ U5 PM sn5 U l
US: a [ UUUS9U l OS U l US: ab US: a\ [ R " [ R" " UR UR$ S9[' XUR S- U5 5 U l g S U l g s snf )Nr r )rt r r rw rO )ry rz r= r r r r" r
r~ group_token
ModuleListrangeGroupViTEncoderLayerlayersr
downsample
Sequentialr} r r group_projector)r rt r= r> r r r! r s r% rz GroupViTStage.__init__ s
. Q !||EKK?L^L^,_ `D #D mm5QV<$X [ TU ] 5 Xl [ UR U l Ub UO UR nUb UO UR nUb UOUn[ R " X#5 U l
[ R " X45 U l g rJ ) ry rz rt r
hidden_act
activation_fnr~ r` r r fc1fc2)r rt r~ r` ra r s r% rz GroupViTMLP.__init__, s| #F$5$5 6%0% [ TU ] UR SS5 5 nUR SS5 $ Nr rO )ry r r )r r r s r% r GroupViTMixerMLP.forwardD s- GOAKK1 - .{{1a r' r )r r r r r r r r s @r% r r C s
! !r' r c ^ \ rS rSrSrU 4S jrS\R S\S\4S jr SS \R S
\R S- S \R S- S \ \R \R S- 4 4S
jjr Sr
U =r$ )r{ iI z=Multi-headed attention from 'Attention Is All You Need' paperc > [ TU ] 5 Xl UR U l UR
U l U R U R - U l U R U R - U R :w a& [ SU R SU R S35 eU R S- U l UR U l [ R " U R U R 5 U l [ R " U R U R 5 U l [ R " U R U R 5 U l [ R " U R U R 5 U l g )Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: r r )ry rz rt r~ r num_attention_heads num_headshead_dimr r^ attention_dropoutr r r r r r out_projr s r% rz GroupViTAttention.__init__L s
++ 33 $..8
==4>> )T^^ ; MdnnM] ^NN#2'
]]D(
// ii? ii? ii? $..$..A
r' rH r bszc UR X2U R U R 5 R SS5 R 5 $ rl )r rq rr r rh )r rH r rv s r% _shapeGroupViTAttention._shape_ s5 {{3 GQQRSUVWbbddr' NrT rX r r c UR 5 u pVnUSLnU R U5 U R - n U( aE U R U R U5 SU5 n
U R U R U5 SU5 n ODU R U R U5 SU5 n
U R U R U5 SU5 n XPR - SU R 4n U R XU5 R " U 6 n U
R " U 6 n
U R " U 6 n U
R S5 n
[ R " XR SS5 5 nUR 5 XPR - Xm4:w a- [ SXPR - Xm4 SUR 5 35 eUbv UR 5 USXm4:w a" [ SUSXm4 SUR 5 35 eUR XPR Xm5 U- nUR XPR - Xm5 n[ R R USS9nUR XPR Xm5 nUR XPR - Xm5 n[ R R! XR U R" S 9n[ R " UU 5 nUR 5 XPR - X`R 4:w a5 [ S
XPR X`R 4 SUR 5 35 eUR XPR X`R 5 nUR SS5 nUR% XVU5 nU R' U5 nUU4$ ) z#Input shape: Batch x Time x ChannelNre r rO z$Attention weights should be of size z , but is z!Attention mask should be of size r )pr z `attn_output` should be of size )rS r r^ rx r r rq rr r r" bmmr r r r r5 r r rY rt )r rT rX r kwargsrv tgt_lenr is_cross_attention query_states
key_states value_states
proj_shapesrc_len attn_weightsattn_weights_reshaped
attn_probs attn_outputs r% r GroupViTAttention.forwardb s= #0"4"4"6 i 2$ > {{=1DJJ> T[[1F%GSQJ ;;t{{3H'I2sSL T[[ %?SIJ ;;t{{='A2sKLNN*B
>
{{<#>CCZP __j1
#((*5 //!$ yy /C/CAq/IJ 3 #7"J J 6nn8Lg7_6` a %%'(*
% ""$a(BB 7a8R7SS\]k]p]p]r\st (,,S..'SVddL',,S>>-A7TL}},,\r,B !- 1 1#~~w X ,11#2FY ]]**<< D2 ||D2 t+D2 % 0 04 7 D2
u||U\\D 00 1
D2 D2r' r{ c ^ \ rS rSrS\4U 4S jjrS\R S\R S\\ S\R 4S jr S r U =r
$ )
rC i rt c < > [ TU ] 5 UR U l [ U5 U l [ R " U R UR S9U l [ U5 U l [ R " U R UR S9U l g rv )
ry rz r~ r r{ self_attnr r} r layer_norm1r r layer_norm2r s r% rz GroupViTEncoderLayer.__init__ sm
++*62 <<F>
')
!0
(( 7
/
0
r' )r r r r r )r r r r r rz r" r r r r r r r r s @r% rC rC sV S 3 S || +,
r' rC c d \ rS rSr% \\S' SrSrSr\ \
S.r \ R " 5 S 5 rSrg )
GroupViTPreTrainedModeli rt groupvit)imagetextT)rT r[ c t U R R n[ U[ R [ R
45 ( aN [ R " UR SUS9 UR b [ R " UR 5 O[ U[ R [ R 45 ( a [ R " UR 5 [ R " UR 5 [ USS5 b` [ R " UR 5 [ R " UR 5 [ R " UR" 5 U R R$ n[ U[& 5 ( a [ R " UR( R SUS- S9 [ R " UR* R SUS- S9 [ R, " UR. [0 R2 " UR. R4 S 5 R7 S5 5 g[ U[8 5 ( a U R R$ nUR: S- S UR R< - S- - U- nUR: S- U- n[ R " UR> R US
9 [ R " UR@ R US
9 [ R " URB R US
9 [ R " URD R US
9 g[ U[F 5 ( a U R R$ nUR RH S- S UR R< - S- - U- nS UR RH - S- U- n[ R " URJ R US
9 [ R " URL R US
9 gg) zInitialize the weightsrC )meanstdN running_meang{Gz?re r' r rO )r )'rt initializer_ranger r r r initnormal_r5 biaszeros_r} BatchNorm1dones_r r running_varnum_batches_trackedinitializer_factorr$ r+ r- copy_r& r" r# rJ r/ r{ r num_hidden_layersr r r rt r r~ re rf )r module
init_rangefactor in_proj_std out_proj_stdfc_stds r%
_init_weights%GroupViTPreTrainedModel._init_weights s [[22
fryy")) 4 5 5 LL Sj A{{& FKK(
r~~ >
?
? KK $ JJv}} %v~t4@ F / /0
6--. F 6 67// f 4 5 5 LL//66Sftm T LL2299RV W JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h
1
2
2[[33F!++T1q6==;Z;Z7Z_c6cdgmmK " , ,d 2f F)
ry rz rt r rA rB r$ depthsr< num_group_tokensnum_output_groupsstagesgradient_checkpointing)r rt ir s r% rz GroupViTVisionEncoder.__init__ s
mm s6== 12
3A ! --*$*$;$;A$>%+%=%=a%@LMPQE)A)A!a%)HWX 3
',#
s AB0rT output_hidden_statesrV return_dictc Ub UOU R R nUb UOU R R nUb UOU R R nU( a SOS nU( a SOS nS n[ U R
5 H= u pU( a XQ4- nU " XU5 n
U
S nU
S nU( d M- U
S c M5 XjS 4- nM? U( a XQ4- nU( d [
S XU4 5 5 $ [ XUS9$ )Nr r r rO c 3 . # U H oc M Uv M
g 7frJ r )r vs r% r 0GroupViTVisionEncoder.forward..0 s g$Uq$Us )last_hidden_staterT r[ )rt rV r r enumerater r r ) r rT r rV r all_hidden_states
all_groupingsr r stage
layer_outputss r% r GroupViTVisionEncoder.forward s 2C1N -TXT_T_TqTq$8$D $++JjJj &1% [ TU ] 5 Xl [ R " [ UR 5 Vs/ s H n[ U5 PM sn5 U l SU l g s snf r] )
ry rz rt r rA rB r rC rD r )r rt r! r s r% rz GroupViTTextEncoder.__init__? sT
mm5QWQiQiKj$kKja%9&%AKj$kl &+# %ls A&NrX r} r c R UnU R H
nU" UU40 UD6nM [ US9$ )a
Args:
inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`):
Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation.
This is useful if you want more control over how to convert `input_ids` indices into associated vectors
than the model's internal embedding lookup matrix.
attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
- 1 for tokens that are **not masked**,
- 0 for tokens that are **masked**.
[What are attention masks?](../glossary#attention-mask)
)r )rD r )r r3 rX r} rT
encoder_layers r% r GroupViTTextEncoder.forwardE sC ( &
![[M ) M ) +
r' )rt r rD rJ )r r r r r r rz r" r r r r r r r r r s @r% r r 6 sV , 1 , /3
t+
+,
r' r c ^ \ rS rSrS\4U 4S jjr\\" SS9\ SS\ R S- S\ R S- S \ R S- S
\ \ S \
4
S jj5 5 5 rS
rU =r$ )GroupViTTextTransformerif rt c > [ TU ] U5 UR n[ U5 U l [ U5 U l [ R " X!R S9U l
UR U l U R 5 g rv )
ry rz r~ r$ r r encoderr r} r final_layer_norm eos_token_id post_initr0 s r% rz GroupViTTextTransformer.__init__g sd
&& 08*62 " Y [ TU ] U5 [ U5 U l U R 5 g rJ )ry rz r
text_modelr r s r% rz GroupViTTextModel.__init__ s&
1&9 r' r c B U R R R $ rJ r r r+ r s r% get_input_embeddings&GroupViTTextModel.get_input_embeddings s ))999r' c 8 XR R l g rJ r )r r s r% set_input_embeddings&GroupViTTextModel.set_input_embeddings s 5:""2r' Nr2 rX r& r} c . U R " SUUUS.UD6$ )a
Examples:
```python
>>> from transformers import CLIPTokenizer, GroupViTTextModel
>>> tokenizer = CLIPTokenizer.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> model = GroupViTTextModel.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt")
>>> outputs = model(**inputs)
>>> last_hidden_state = outputs.last_hidden_state
>>> pooled_output = outputs.pooler_output # pooled (EOS token) states
```r2 rX r& r r )r r2 rX r& r} s r% r GroupViTTextModel.forward s- .
)%
r' r r9 )r r r r r r r rz r Moduler r r r" r r r r r r r r r s @r% r r s 1 :bii :; *..2,0
<<$&
t+
llT)
+,
+ +
r' r c
^ \ rS rSrS\4U 4S jjr\ S S\R S- S\ S- S\ S- S\ S- S \
\ - 4
S
jj5 r S r
U =r$ )
GroupViTVisionTransformeri rt c > [ TU ] 5 Xl UR n[ U5 U l [
U5 U l [ R " X!R S9U l g rv ) ry rz rt r~ r r r r r r} r r r0 s r% rz "GroupViTVisionTransformer.__init__ sL
&& 26:,V4 i5J5JKr' Nr r rV r r c Ub UOU R R nUb UOU R R nUb UOU R R nUc [ S5 eU R U5 nU R
UUUUS9nUS nU R U5 nUR SS9nU( d Xx4USS - $ [ UUUR UR S9$ )Nz You have to specify pixel_values)rT r rV r r r r )r r rT r[ ) rt rV r r r r r r r r rT r[ ) r r r rV r rT r r r s r% r !GroupViTVisionTransformer.forward s 2C1N -TXT_T_TqTq$8$D $++JjJj &1% [ TU ] U5 [ U5 U l U R 5 g rJ )ry rz r vision_modelr r s r% rz GroupViTVisionModel.__init__ s'
5f = r' r c B U R R R $ rJ )r r r r s r% r (GroupViTVisionModel.get_input_embeddings# s ++<<>> from PIL import Image
>>> import httpx
>>> from io import BytesIO
>>> from transformers import AutoProcessor, GroupViTVisionModel
>>> processor = AutoProcessor.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> model = GroupViTVisionModel.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> with httpx.stream("GET", url) as response:
... image = Image.open(BytesIO(response.read()))
>>> inputs = processor(images=image, return_tensors="pt")
>>> outputs = model(**inputs)
>>> last_hidden_state = outputs.last_hidden_state
>>> pooled_output = outputs.pooler_output # pooled CLS states
```r rV r r r )r r rV r r r} s r% r GroupViTVisionModel.forward& s( > % /!5# !
r' r r )r r r r r r main_input_namer r rz r r r r" r r r r r r r r s @r% r r s $O! 3 =&= = 26)-,0#' #
''$.#
$;#
#Tk #
D[ #
+ +#
#
r' r c ^ \ rS rSr% \\S' S\4U 4S jjr\\ SS\ R S\ R S- S\ R S- S\ \ S \
\- 4
S
jj5 5 r\\S \ R S\ \ S \
\- 4S j5 5 r\\ SS\ R" S- S \ R$ S- S\ R S- S\ R" S- S
\S- S\S- S\S- S\S- S\ \ S \
\- 4S jj5 5 rSrU =r$ )
GroupViTModeliM rt c > > [ TU ] U5 [ UR [ 5 ( d" [ S[
UR 5 S35 e[ UR [ 5 ( d" [ S[
UR 5 S35 eUR nUR nUR U l UR U l
UR U l UR U l
[ U5 U l [! U5 U l [$ R& " [$ R( " U R U R SS9[$ R* " U R 5 [$ R, " SS9[$ R( " U R U R SS95 U l [$ R& " [$ R( " U R U R SS9[$ R* " U R 5 [$ R, " SS9[$ R( " U R U R SS95 U l [$ R2 " [4 R6 " U R8 R: 5 5 U l U R? 5 g )NzOconfig.text_config is expected to be of type GroupViTTextConfig but is of type .zSconfig.vision_config is expected to be of type GroupViTVisionConfig but is of type T)r )inplace) ry rz r text_configr TypeErrortype
vision_configr projection_dimprojection_intermediate_dimr~ text_embed_dimvision_embed_dimr r r r r rF r r ReLUvisual_projectiontext_projectionr r" rH rt logit_scale_init_value logit_scaler )r rt r r r s r% rz GroupViTModel.__init__Q s
&,,.@AA++,-Q0
&..0DEE--./q2
(( ,,
$ 3 3+1+M+M( ) 5 5 - 9 9 1+> 5m D!# IId++T-M-MTX Y NN4;; < GGD ! IId668K8KRV W "
"}} IId))4+K+KRV W NN4;; < GGD ! IId668K8KRV W
<< T[[5W5W(XY
r' Nr2 rX r& r} r c x U R " SUUUSS.UD6nUR nU R U5 Ul U$ )a
Examples:
```python
>>> import torch
>>> from transformers import CLIPTokenizer, GroupViTModel
>>> model = GroupViTModel.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> tokenizer = CLIPTokenizer.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt")
>>> with torch.inference_mode():
... text_features = model.get_text_features(**inputs)
```T)r2 rX r& r r )r r r )r r2 rX r& r} text_outputsr s r% get_text_features GroupViTModel.get_text_features| sS . 48?? 4
)% 4
4
%22
%)%9%9-%H "r' r c p U R " U4SS0UD6nU R UR 5 Ul U$ )a"
Examples:
```python
>>> import torch
>>> from transformers import AutoProcessor, GroupViTModel
>>> from transformers.image_utils import load_image
>>> model = GroupViTModel.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> processor = AutoProcessor.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = load_image(url)
>>> inputs = processor(images=image, return_tensors="pt")
>>> with torch.inference_mode():
... image_features = model.get_image_features(**inputs)
```r T)r r r )r r r} vision_outputss r% get_image_features GroupViTModel.get_image_features s? 4 6:5F5F|5pae5pio5p'+'='=n>Z>Z'[$ r' return_lossrV r output_segmentationc Ub UOU R R nUb UOU R R nU( a SnUb UOU R R nU R UUUSS9n
U R
" S UUUS.U D6n U
R n U R U 5 n U R n
U R U
5 n
XR SSS9- n XR SSS9- n
U R R 5 n[ R " XR 5 5 U- nUR 5 nSnU( Gag U
R nU R UR! SUR" S 5 5 nU
R$ n[' UUR" SS 5 nUUR SSS9- n[ R " UU
R 5 5 U- nUR! U R" S SU
R" S 5 R) SSS 5 nUR! UR" S UR" S S5 n[ R " UU5 U- nUR! UR" S UR" S UR" S UR" S
5 nSnU( a [+ U5 n[- UUUUU
U U U
S 9$ )
a
return_loss (`bool`, *optional*):
Whether or not to return the contrastive loss.
output_segmentation (`bool`, *optional*):
Whether or not to return the segmentation logits.
Examples:
```python
>>> from PIL import Image
>>> import httpx
>>> from io import BytesIO
>>> from transformers import AutoProcessor, GroupViTModel
>>> model = GroupViTModel.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> processor = AutoProcessor.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> with httpx.stream("GET", url) as response:
... image = Image.open(BytesIO(response.read()))
>>> inputs = processor(
... text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True
... )
>>> outputs = model(**inputs)
>>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score
>>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities
```NTr r re r rO r r r )r r r r r r r r r )rt rV r' r r r r r r normr expr" matmultr rY rJ r[ rp rg r- r )r r2 r rX r& r& rV r r' r} r# r r r r r r
seg_logitsimage_group_embedsr[ groupinglogits_per_image_groupflatten_groupingr s r% r GroupViTModel.forward s X 2C1N -TXT_T_TqTq#6#B HgHg $ $8$D $++JjJj
**% /!5 +
48?? 4
)%4
4
&33 --l; "00 **;7 $&7&7B&7&MM !$4$4T$4$JJ &&**, ,,{NN4DE S*,,.
"0!A!A !%!7!78J8R8RSUWiWoWoprWs8t!u '22J3J @R@RSTSU@VWH "46H6M6MRT^b6M6c!c %*\\2Dkmmo%VYd%d "%;%C%C ""1%r;+<+>RSCTVX Y &<>NOR]]J#++ #Z%5%5a%8(..:KX^^\]M^J .?D" - + *#% * .
r' ) r r r r r r r r r r )NNNNNNNN)r r r r r r rz r r r" r r r r r r r$ r: r r r r r r r s @r% r r M s )~ )V /3,0 << t+ llT)
+,
+ +
B ll +,
+ + : .215.204#')-,0+/|
##d*|
''$.|
t+ |
&&- |
D[
|
$;|
#Tk|
"D[|
+,|
$ $|
|
r' r )r r r r )r Fre r )Mr collections.abcr dataclassesr typingr numpyrW r" r r r activationsr
masking_utilsr modeling_layersr
modeling_outputsr r modeling_utilsr
processing_utilsr utilsr r r r r r
utils.genericr utils.output_capturingr configuration_groupvitr r r
get_loggerr loggerr r& r- rV r? floatr rM rc rp r rr r r r r r r$ r< r r r{ rC r r r r r r r r __all__r r' r% rF sa ! & ! / 9 K - & j j 7 5 \ \
H %
`U\\ `ell `
-ELL -U\\ - C 5<< e t RU _d_k_k ,<:
"))
- bii - `4+")) 4+n -
+ -
-
` bii DGryy GV%RYY %PZBII Zz ")) 0!{ ! ]2 ]2B 5 B -=o -= -=`7
BII 7
t-
")) -
`J
5 J
Z,
/ ,
^2
2
j3
1 3
l n
+ n
n
b cr'