ZjeSrSSKJr SSKrSSKJr SSKJr SSKJ r SSK J r SS K J r JrJr SS KJrJr SS KJr SS KJrJrJrJr SS KJrJr SSKJr SSKJ r \RB"\"5r#"SS\RH5r%"SS\RH5r&S7S\RHS\RNS\RNS\RNS\RNS-S\(S-S\(S\\4Sjjr)"SS\RH5r*"S S!\RH5r+"S"S#\RH5r,"S$S%\RH5r-"S&S'\RH5r."S(S)\ 5r/"S*S+\RH5r0"S,S-\RH5r1\"S.S/\55r2\"S0S1\255r3\"S2S39"S4S5\255r4/S6Qr5g)8zPyTorch ViViT model.)CallableN)nn)initialization)ACT2FN)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringlogging torch_int)can_return_tuplemerge_with_config_defaults)capture_outputs) VivitConfigcv^\rSrSrSrS\4U4SjjrS S\RS\ S\R4Sjjr S r U=r $) VivitTubeletEmbeddings$az Construct Vivit Tubelet embeddings. This module turns a batch of videos of shape (batch_size, num_frames, num_channels, height, width) into a tensor of shape (batch_size, seq_len, hidden_size) to be consumed by a Transformer encoder. The seq_len (the number of patches) equals (number of frames // tubelet_size[0]) * (height // tubelet_size[1]) * (width // tubelet_size[2]). configc>[TU]5 URUlURUlURUlURUR S-URUR S--URUR S--UlURUl[R"URURURURS9Ul g)Nrr) kernel_sizestride) super__init__ num_frames image_size tubelet_size patch_size num_patches hidden_size embed_dimrConv3d num_channels projectionselfr __class__s y/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/vivit/modeling_vivit.pyr!VivitTubeletEmbeddings.__init__/s  ++ ++ -- __ 2 2$//!"44 6$//!"44 6   ++))   !3!3ATAT]c]p]p  pixel_valuesinterpolate_pos_encodingreturnc bURup4pVnU(dPX`R:wdXpR:wa2[SUSUSURSSURSS3 5eURSSSSS 5nUR U5nUR S5R SS5nU$) NzImage image size (*z) doesn't match model (rrz).rr)shaper# ValueErrorpermuter+flatten transpose) r-r2r3 batch_sizer"r*heightwidthxs r/forwardVivitTubeletEmbeddings.forward?s>J>P>P; e'V-F%SbSbJb$VHAeW4KDOO\]L^K__`aeapapqras`ttvw  $++Aq!Q: OOL ) IIaL " "1a (r1)r(r#r"r&r%r+F) __name__ __module__ __qualname____firstlineno____doc__rr!torchTensorboolrA__static_attributes__ __classcell__r.s@r/rr$s> { ELLD]b]i]ir1rc^\rSrSrSrS\4U4SjjrS\RS\ S\ S\R4S jr SS \RS \ S\R4S jjr S r U=r$)VivitEmbeddingsPz| Vivit Embeddings. Creates embeddings from a video using VivitTubeletEmbeddings, adds CLS token and positional embeddings. rc>[TU]5 [R"[R "SSUR 55Ul[U5Ul [R"[R "SURRS-UR 55Ul [R"UR5UlURSSUlXlg)Nr)r r!r ParameterrIzerosr' cls_tokenrpatch_embeddingsr&position_embeddingsDropouthidden_dropout_probdropoutr$r%rr,s r/r!VivitEmbeddings.__init__Ws ekk!Q8J8J&KL 6v >#%<< KK400<r?r4cURSS- nURRSS- n[RR 5(dXE:XaX#:Xa UR$URSS2SS24nURSS2SS24nURSnX R S-n X0R S-n [ US-5n URSXU5nURSSSS5n[RRUX4SS S 9nURSSSS5RSSU5n[R"Xg4SS 9$) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNrg?rrbicubicF)sizemode align_cornersdim)r8rWrIjit is_tracingr%rreshaper:r functional interpolateviewcat) r-r\r>r?r& num_positionsclass_pos_embedpatch_pos_embedrd new_height new_widthsqrt_num_positionss r/r3(VivitEmbeddings.interpolate_pos_encodinges]!&&q)A- 0066q9A= yy##%%+*F6?++ +221bqb59221ab59r"q11 __Q// &}c'9:)11!5G]`a)11!Q1=--33 ( 4 *11!Q1=BB1b#Nyy/;CCr1r2r3c"URup4pVnURXS9nURRUSS/5n [R "X4SS9nU(aXR XU5-nOXR-nURU5nU$)Nr3rrc) r8rVrUtilerIrkr3rWrZ) r-r2r3r=r"r*r>r?r\ cls_tokenss r/rAVivitEmbeddings.forwards>J>P>P; e**<*k ^^((*a);< YY 7Q?  $#&C&CJX]&^^J#&>&>>J\\*- r1)rUrrZrVr%rWrC)rDrErFrGrHrr!rIrJintr3rKrArLrMrNs@r/rPrPPsq { &D5<<&D&DUX&D]b]i]i&DPELLD]b]i]ir1rPmodulequerykeyvalueattention_maskscalingrZkwargsc UcURS5S-n[R"XRSS55U-nUbX-n[R R USS9n[R RXURS9n[R"X5n U RSS5R5n X4$)Nr^rrrc)ptrainingr) r`rImatmulr<rrhsoftmaxrZr contiguous) ryrzr{r|r}r~rZr attn_weights attn_outputs r/eager_attention_forwardrs**R.D(<<}}Q':;gEL!#4 ==((2(>L==((6??([L,,|3K''1-88:K  $$r1c^\rSrSrS\4U4SjjrS\RS\\ S\ \R\R44Sjr Sr U=r $) VivitSelfAttentionrc0>[TU]5 URUR-S:wa7[ US5(d&[ SURSURS35eXlURUl[URUR- 5UlURUR-Ul URUl URS-Ul SUl [R"URURUR S9Ul[R"URURUR S9Ul[R"URURUR S9Ulg) Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .rF)bias)r r!r'num_attention_headshasattrr9rrxattention_head_size all_head_sizeattention_probs_dropout_prob dropout_probr~ is_causalrLinearqkv_biasrzr{r|r,s r/r!VivitSelfAttention.__init__sG     : : :a ?PVXhHiHi"6#5#5"67334A7   #)#=#= #&v'9'9FV>VV HH]+00)<FFq!L jj/44i@JJ1aP jj/44i@JJ1aP (?(M(M KK , ,.E) *=      * nnLL#}}C$2C2C *  * & #0"4"4"6s";t?Q?Q>S"S%--.EF --r1) rrrrrr{rrzr~r|)rDrErFrGrr!rIrJrrtuplerArLrMrNs@r/rrsS]{](.||.+,. u||U\\) * ..r1rc^\rSrSrSrS\4U4SjjrS\RS\RS\R4Sjr S r U=r $) VivitSelfOutputz The residual connection is defined in VivitLayer instead of here (as is the case with other models), due to the layernorm applied before each block. rc>[TU]5 [R"URUR5Ul[R "UR5UlgN) r r!rrr'denserXrYrZr,s r/r!VivitSelfOutput.__init__sB YYv1163E3EF zz&"<"<= r1r input_tensorr4cJURU5nURU5nU$rrrZr-rrs r/rAVivitSelfOutput.forwards$ =1  ]3 r1r rDrErFrGrHrr!rIrJrArLrMrNs@r/rrsB >{> U\\RWR^R^r1rct^\rSrSrS\4U4SjjrS\RS\\ S\R4Sjr Sr U=r $) VivitAttentionircb>[TU]5 [U5Ul[ U5Ulgr)r r!r attentionroutputr,s r/r!VivitAttention.__init__s& +F3%f- r1rrr4c RUR"U40UD6up4URX15nU$rrr)r-rrself_attn_output_rs r/rAVivitAttention.forward s/ #nn]EfE-= r1r) rDrErFrGrr!rIrJrrrArLrMrNs@r/rrsC.{. ||+,  r1rcj^\rSrSrS\4U4SjjrS\RS\R4SjrSr U=r $)VivitIntermediateirc^>[TU]5 [R"URUR 5Ul[R"UR5Ul [UR[5(a[URUlgURUlgr)r r!rrr'intermediate_sizerrXrYrZ isinstance hidden_actstrrintermediate_act_fnr,s r/r!VivitIntermediate.__init__sv YYv1163K3KL zz&"<"<= f'' - -'-f.?.?'@D $'-'8'8D $r1rr4clURU5nURU5nURU5nU$r)rrrZ)r-rs r/rAVivitIntermediate.forward!s4 =1 00?  ]3 r1)rrZr rDrErFrGrr!rIrJrArLrMrNs@r/rrs/9{9U\\ellr1rc^\rSrSrS\4U4SjjrS\RS\RS\R4SjrSr U=r $) VivitOutputi)rc>[TU]5 [R"URUR 5Ul[R"UR5Ul gr) r r!rrrr'rrXrYrZr,s r/r!VivitOutput.__init__*sB YYv779K9KL zz&"<"<= r1rrr4cRURU5nURU5nX-nU$rrrs r/rAVivitOutput.forward/s, =1  ]3 %4 r1rrrNs@r/rr)s=>{> U\\RWR^R^r1rcn^\rSrSrSrS\4U4SjjrS\RS\R4Sjr Sr U=r $) VivitLayeri6zNThis corresponds to the EncoderBlock class in the scenic/vivit implementation.rcj>[TU]5 URUlSUl[ U5Ul[ U5Ul[U5Ul [R"URURS9Ul[R"URURS9Ulg)Nreps)r r!chunk_size_feed_forward seq_len_dimrrr intermediaterrr LayerNormr'layer_norm_epslayernorm_beforelayernorm_afterr,s r/r!VivitLayer.__init__9s '-'E'E$'/-f5!&) " V-?-?VEZEZ [!||F,>,>FDYDYZr1rr4cURU5nURU5nX1-nURU5nURU5nUR XA5nU$r)rrrrr)r-rhidden_states_normattention_output layer_outputs r/rAVivitLayer.forwardCse!22=A>>*<=)8 ++M: ((6 {{<? r1)rrrrrrrrrNs@r/rr6s4X[{[U\\ellr1rcV^\rSrSrS\4U4SjjrS\RS\4Sjr Sr U=r $) VivitEncoderiTrc>[TU]5 Xl[R"[ UR 5Vs/sHn[U5PM sn5UlSUl gs snfNF) r r!rr ModuleListrangenum_hidden_layersrlayergradient_checkpointing)r-rrr.s r/r!VivitEncoder.__init__UsR  ]]fF^F^@_#`@_1Jv$6@_#`a &+#$asA&rr4c`[UR5H up#U"U5nM [US9$)N)last_hidden_state) enumeraterr )r-ri layer_modules r/rAVivitEncoder.forward[s.(4OA(7M 5??r1)rrr) rDrErFrGrr!rIrJr rArLrMrNs@r/rrTs/,{, @U\\@o@@r1rcj^\rSrSrS\4U4SjjrS\RS\R4SjrSr U=r $) VivitPooleribrc>[TU]5 [R"URUR5Ul[R "5Ulgr)r r!rrr'rTanh activationr,s r/r!VivitPooler.__init__cs9 YYv1163E3EF '')r1rr4c\USS2S4nURU5nURU5nU$)Nr)rr)r-rfirst_token_tensor pooled_outputs r/rAVivitPooler.forwardhs6+1a40 #56  6 r1)rrrrNs@r/rrbs/${$ U\\ellr1rc^\rSrSr%\\S'SrSrSrSr S/r Sr Sr Sr Sr\\S.r\R&"5U4S j5rS rU=r$) VivitPreTrainedModeliqrvivitr2videoTr)r attentionsc>[TU]U5 [U[5(aA[R "UR 5 [R "UR5 gg)zInitialize the weightsN)r _init_weightsrrPinitzeros_rUrW)r-ryr.s r/r"VivitPreTrainedModel._init_weightssH f% fo . . KK(( ) KK22 3 /r1)rDrErFrGr__annotations__base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backendrr_can_record_outputsrIno_gradrrLrMrNs@r/rrqsg $O&*#%N"&#(  ]]_44r1rc ^\rSrSrSS\S\4U4SjjjrSr\\ "SS9\ SS \ RS-S \S \ \S \4S jj555rSrU=r$) VivitModeliradd_pooling_layerc>[TU]U5 Xl[U5Ul[ U5Ul[R"URURS9Ul U(a [U5OSUl UR5 g)z^ add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer rN)r r!rrPr\rencoderrrr'r layernormrpooler post_init)r-rrr.s r/r!VivitModel.__init__si   )&1#F+ f&8&8f>S>ST->k&)D  r1c.URR$r)r\rV)r-s r/get_input_embeddingsVivitModel.get_input_embeddingss///r1F)tie_last_hidden_statesNr2r3rr4c Uc [S5eURXS9nURU5nURnUR U5nUR bUR U5OSn[ XgS9$)a( Examples: ```python >>> import av >>> import numpy as np >>> from transformers import VivitImageProcessor, VivitModel >>> from huggingface_hub import hf_hub_download >>> np.random.seed(0) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`list[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # video clip consists of 300 frames (10 seconds at 30 FPS) >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample 32 frames >>> indices = sample_frame_indices(clip_len=32, frame_sample_rate=1, seg_len=container.streams.video[0].frames) >>> video = read_video_pyav(container=container, indices=indices) >>> image_processor = VivitImageProcessor.from_pretrained("google/vivit-b-16x2-kinetics400") >>> model = VivitModel.from_pretrained("google/vivit-b-16x2-kinetics400") >>> # prepare video for the model >>> inputs = image_processor(list(video), return_tensors="pt") >>> # forward pass >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 3137, 768] ```Nz You have to specify pixel_valuesrt)r pooler_output)r9r\rrrrr )r-r2r3rembedding_outputencoder_outputssequence_outputrs r/rAVivitModel.forwardsuh  ?@ @?? Note that it's possible to fine-tune ViT on higher resolution images than the ones it has been trained on, by setting `interpolate_pos_encoding` to `True` in the forward of the model. This will interpolate the pre-trained position embeddings to the higher resolution. ) custom_introc ^\rSrSrS\4U4Sjjr\\S S\RS-S\RS-S\ S\ \ S \4 S jj55rS rU=r$) VivitForVideoClassificationirc.>[TU]U5 URUl[USS9UlURS:a+[ R "URUR5O[ R"5Ul UR5 g)NF)rr) r r! num_labelsrrrrr'Identity classifierrr,s r/r!$VivitForVideoClassification.__init__ss   ++%@ OUN_N_bcNc"))F$6$68I8IJikititiv r1Nr2labelsr3rr4c UR"U4SU0UD6nURnURUSS2SSS245nSnUbUR"X'UR40UD6n[ UUUR URS9$)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> import av >>> import numpy as np >>> import torch >>> from transformers import VivitImageProcessor, VivitForVideoClassification >>> from huggingface_hub import hf_hub_download >>> np.random.seed(0) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`list[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # video clip consists of 300 frames (10 seconds at 30 FPS) >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample 32 frames >>> indices = sample_frame_indices(clip_len=32, frame_sample_rate=4, seg_len=container.streams.video[0].frames) >>> video = read_video_pyav(container=container, indices=indices) >>> image_processor = VivitImageProcessor.from_pretrained("google/vivit-b-16x2-kinetics400") >>> model = VivitForVideoClassification.from_pretrained("google/vivit-b-16x2-kinetics400") >>> inputs = image_processor(list(video), return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) ... logits = outputs.logits >>> # model predicts one of the 400 Kinetics-400 classes >>> predicted_label = logits.argmax(-1).item() >>> print(model.config.id2label[predicted_label]) LABEL_116 ```r3Nr)losslogitsrr)rrr2 loss_functionrr rr) r-r2r4r3routputsr)r7r6s r/rA#VivitForVideoClassification.forwardsx$(:: $ 3K$ OU$ "33Aq!9:  %%fdkkLVLD$!//))   r1)r2r0r)NNF)rDrErFrGrr!rrrIr+ LongTensorrKrrr rArLrMrNs@r/r.r.s { 26*.). i ''$.i   4'i #' i +, i  i i r1r.)rrr.)Nr)6rHcollections.abcrrIrrr activationsrmodeling_layersrmodeling_outputsr r r modeling_utilsr r processing_utilsrutilsrrrr utils.genericrrutils.output_capturingrconfiguration_vivitr get_loggerrDloggerModulerrPrJfloatrrrrrrrrrrrr.__all__r r1r/rLs$ &!9bbF&KKI5,   H %)RYY)XLbiiLl!% II% <<% % << % LL4' % T\ %%'(%:4.4.pbii$ RYY   $ ")) +< @299 @ ")) 4?442rj%rjrjj  x "6x  x v Pr1