Zj.SSKrSSKrSSKJr SSKJr SSKrSSKrSSK J s J r SSKJ r J r SSKJr SSKJr SSKJrJrJr SS KJr SS KJrJr SS KJr SS KJrJ r J!r! SS K"J#r# SSK$J%r% SSK&J'r' \"5(aSSK(J)r) \!"5(a SSK*J+r+ SSK,J-r- "SS\ R\5r/"SS\ R\5r0"SS\ R\5r1"SS\ R\5r2SRS\ R\S\RS\RS\RS \RS-S!\3S"\34S#jjr4"S$S%\ R\5r5SSS&\RS'\3S(\6S)\R4S*jjr7"S+S,\ R\5r8"S-S.\ R\5r9"S/S0\5r:"S1S2\ R\5r;\\ "S3S49"S5S6\555r<STS7\RS8\RS)\R4S9jjr=S:\ S;\ S)\ 4S<jr>S:\RS;\RS)\R4S=jr?"S>S?\ R\5r@S:\ S;\ S@\AS)\ 4SAjrBS:\RS;\RS@\AS)\R4SBjrC"SCSD\ R\5rD\ "SESF\55rE"SGSH\ R5rG"SISJ\ R\5rH"SKSL\ R\5rI"SMSN\ R\5rJ\ "SOS49"SPSQ\E55rKSFSQ/rLg)UN)Callable) dataclass)Tensornn)initialization)ACT2FN) ModelOutputis_scipy_availablerequires_backends)GradientCheckpointingLayer)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringis_accelerate_available)merge_with_config_defaults)capture_outputs) VideomtConfig)linear_sum_assignment) PartialState)reducecf^\rSrSrSrU4SjrS\RS\R4SjrSr U=r $)VideomtPatchEmbeddings3z This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. c>[TU]5 URURp2URUR pT[ U[RR5(aUOX"4n[ U[RR5(aUOX34nUSUS-USUS--nX lX0lX@lX`l [R"XEX3S9Ul g)Nrr kernel_sizestride)super__init__ image_size patch_size num_channels hidden_size isinstance collectionsabcIterable num_patchesrConv2d projection)selfconfigr$r%r&r'r, __class__s }/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/videomt/modeling_videomt.pyr#VideomtPatchEmbeddings.__init__:s !'!2!2F4E4EJ$*$7$79K9Kk#-j+//:R:R#S#SZZdYq #-j+//:R:R#S#SZZdYq !!} 1 5*Q-:VW=:XY $$(&))L:i pixel_valuesreturnc4URSnX R:wa[SURSUS35eURURR R S9nUR U5RS5RSS5nU$)NrzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .dtype) shaper& ValueErrortor.weightr:flatten transpose)r/r5r& embeddingss r2forwardVideomtPatchEmbeddings.forwardIs#))!, ,, ,!../yaI  $T__-C-C-I-IJ __\2::1=GG1M r4)r$r&r,r%r.) __name__ __module__ __qualname____firstlineno____doc__r#torchrrC__static_attributes__ __classcell__r1s@r2rr3s. j ELL U\\  r4rc^\rSrSrSrS\SS4U4SjjrS S\RS\RS-S\R4S jjr S r U=r $) VideomtEmbeddingsVzE Construct the CLS token, mask token, position and patch embeddings. r0r6Nc6>[TU]5 XlURUl[R "[ R"SSUR55Ul [R "[ R"SURUR55Ul [U5UlURRn[R "UR"5UlSUR-Ul[R("X!R5UlUR-S[ R."U5R1S5SS9 [R "[ R"SSUR55Ulg)Nr position_idsrF) persistent)r"r#r0r%r ParameterrJrandnr' cls_tokenzerosnum_register_tokensregister_tokensrpatch_embeddingsr,Dropouthidden_dropout_probdropoutnum_prefix_tokens Embeddingposition_embeddingsregister_bufferarangeexpand mask_token)r/r0r,r1s r2r#VideomtEmbeddings.__init__[s   ++ekk!Q8J8J&KL!||EKK6;U;UW]WiWi,jk 6v >++77 zz&"<"<= !"V%?%?!?#%<< =O=O#P  ^U\\+-F-M-Mg-Vchi,,u{{1a9K9K'LMr4r5bool_masked_poscURS:Xa=URup4pVnURX4-XVU5nUbURX4-S5nO2Ub/URS:aURURSS5nURSnURU5nUbXUR UR [ RS9RS5n [ R"XRU5nURRUSS5n URRUSS5n XRUR5-n[ R "XU/SS9nUR#U5nU$)NrTr;r)devicer:rdim)ndimr<reshaper\r>rkrJbool unsqueezewhererfrXrer[rbrRcatr_) r/r5rh batch_size num_framesr&heightwidthrBmask cls_tokensr[s r2rCVideomtEmbeddings.forwardksa    !BNBTBT ?JL%'// 0G_deL*"1"9"9*:QSU"V  (_-A-AA-E-55o6K6KA6NPRSO!''* **<8  &"%%Z->->ejj%Q[[\^_DT??JGJ^^**:r2> ..55j"bI":":4;L;L"MM YY ZHaP \\*- r4) rXr0r_rfr`r\r%rbr[N rErFrGrHrIrr#rJrrCrKrLrMs@r2rOrOVsRN}NN ELL5<[TU]5 UR=p#[URUR-5n[ R "X$SS9Ul[UR[5(a[URUl OURUl [ R "XCSS9Ul g)NTbias)r"r#r'int mlp_ratiorLinearfc1r( hidden_actstrr activationfc2r/r0 in_features out_featureshidden_featuresr1s r2r#VideomtMLP.__init__s %+%7%77 f0063C3CCD99[E f'' - -$V%6%67DO$//DO99_Fr4 hidden_stateclURU5nURU5nURU5nU$r{)rrrr/rs r2rCVideomtMLP.forwards2xx - |4 xx - r4)rrrr6N rErFrGrHr#rJrrCrKrLrMs@r2r~r~s) GELLU\\r4r~cf^\rSrSrSU4SjjrS\R S\R 4SjrSrU=r $)VideomtGatedMLPr6c$>[TU]5 UR=p#[URUR-5n[US-S- 5S-S-S-n[ R "USU-SS9Ul[ R "XCSS9UlgNr;rTr r"r#r'rrrr weights_in weights_outrs r2r#VideomtGatedMLP.__init__ %+%7%77 f0063C3CCD2Q67!;AAE))K_1D4P99_Nr4rcURU5nURSSS9up#[RR U5U-nUR U5$Nr;rTrlrchunkr functionalsilurr/rx1x2hiddens r2rCVideomtGatedMLP.forwardQ|4 ##A2#.##B'",''r4rrrrrMs@r2rr)O(ELL(U\\((r4rmodulequerykeyvalueattention_maskscalingr_c [R"XRSS55U-nUbX-n[RR US[R S9RUR5n[RRXURS9n[R"X5n U RSS5R5n X4$)NrT)rmr:)ptrainingrr;) rJmatmulrArrsoftmaxfloat32r>r:r_r contiguous) rrrrrrr_kwargs attn_weights attn_outputs r2eager_attention_forwardrs<<}}R'<=GL!#4 ==((2U]](SVVW\WbWbcL==((6??([L,,|3K''1-88:K  $$r4c ^\rSrSrSrU4SjrS S\RS\RS-S\\R\RS-44Sjjr S r U=r $) VideomtAttentionz=Multi-headed attention from 'Attention Is All You Need' paperc >[TU]5 XlURUlUR UlURUR -UlURUR -UR:wa&[SURSUR S35eURS-Ul URUl SUl [R"URUR5Ul[R"URUR5Ul[R"URUR5Ul[R"URUR5Ulg)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gF)r"r#r0r' embed_dimnum_attention_heads num_headshead_dimr=scaleattention_dropoutr_ is_causalrrk_projv_projq_projout_projr/r0r1s r2r#VideomtAttention.__init__s  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( // ii? ii? ii?  $..$..A r4N hidden_statesrr6c URSSn/UQSPURP7nURU5RU5R SS5nUR U5RU5R SS5nUR U5RU5R SS5n[R"URR[5n U "UUUUUURURUR(dSO URS9upU R "/UQSP76R#5n UR%U 5n X4$)z#Input shape: Batch x Time x ChannelNrTrr;)rrr_)r<rrviewrArrr get_interfacer0_attn_implementationrrrrr_rorr) r/rrr input_shape hidden_shapequerieskeysvaluesattention_interfacerrs r2rCVideomtAttention.forwards6$))#2. 88b8$--8 ++m,11,?II!QO{{=)..|<FFq!L]+00>HHAN(?(M(M KK , ,.E) %8     nnJJ#}}C$,, % ! "));;;;FFH mmK0 ((r4) r0r_rrrrrrrrrr{) rErFrGrHrIr#rJrtuplerCrKrLrMs@r2rrs[GB./3!)||!) t+!) u||U\\D00 1 !)!)r4rinput drop_probrr6cUS:XdU(dU$SU- nURS4SURS- --nU[R"X@RUR S9-nUR 5 URU5U-nU$)z[ Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). rrr)rr:rk)r<rnrJrandr:rkfloor_div)rrr keep_probr< random_tensoroutputs r2 drop_pathrs Cx I I [[^  Q 7 7E 5 ELL YYM YYy !M 1F Mr4c^\rSrSrSrS S\S-SS4U4SjjjrS\RS\R4Sjr S\ 4S jr S r U=r $) VideomtDropPathi zXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).Nrr6c.>[TU]5 Xlgr{)r"r#r)r/rr1s r2r#VideomtDropPath.__init__ s "r4rcB[XRUR5$r{)rrrr/rs r2rCVideomtDropPath.forwards FFr4c SUR3$)Nzp=rr/s r2 extra_reprVideomtDropPath.extra_reprsDNN#$$r4rr{)rErFrGrHrIfloatr#rJrrCrrrKrLrMs@r2rr sQb#%$,#$##GU\\GellG%C%%r4rcf^\rSrSrSU4SjjrS\R S\R 4SjrSrU=r $)VideomtSwiGLUFFNir6c$>[TU]5 UR=p#[URUR-5n[US-S- 5S-S-S-n[ R "USU-SS9Ul[ R "XCSS9Ulgrrrs r2r#VideomtSwiGLUFFN.__init__rr4rcURU5nURSSS9up#[RR U5U-nUR U5$rrrs r2rCVideomtSwiGLUFFN.forward"rr4rrrrMs@r2rrrr4rc^\rSrSrSrS\SS4U4SjjrS S\RS\RS-S\R4S jjr S r U=r $) VideomtLayeri)zCThis corresponds to the Block class in the original implementation.r0r6Nc>[TU]5 [R"URUR S9Ul[U5Ul[U5Ul URS:a[UR5O[R"5Ul[R"URUR S9UlUR (a[#U5UlO['U5Ul[U5Ulg)Nepsr)r"r#r LayerNormr'layer_norm_epsnorm1r attentionVideomtLayerScale layer_scale1drop_path_raterIdentityrnorm2use_swiglu_ffnrmlpr~ layer_scale2rs r2r#VideomtLayer.__init__,s \\&"4"4&:O:OP )&1-f5CICXCX[^C^)>)>?dfdododq\\&"4"4&:O:OP  '/DH!&)DH-f5r4rrc&URU5nURX25upEURU5nURU5U-nUR U5nUR U5nUR U5nURU5U-nU$r{)rrr rr rr)r/rrhidden_states_normself_attention_output_ layer_outputs r2rCVideomtLayer.forward<s "ZZ 6#'>>2D#U  $ 1 12G H'<= M zz-0 xx - ((6 ~~l3mC r4)rrr rrrr r{r|rMs@r2rr)sUM6}66&/3|| t+  r4rcf^\rSrSrSU4SjjrS\R S\R 4SjrSrU=r $)riSr6c>[TU]5 [R"UR[ R "UR5-5Ulgr{) r"r#rrVlayerscale_valuerJonesr'lambda1rs r2r#VideomtLayerScale.__init__Ts8 ||F$;$;ejjI[I[>\$\] r4rcXR-$r{rrs r2rCVideomtLayerScale.forwardXsll**r4rrrrMs@r2rrSs)^+ELL+U\\++r4ra Class for outputs of [`VideomtForUniversalSegmentationOutput`]. This output can be directly passed to [`~VideomtVideoProcessor.post_process_semantic_segmentation`] or [`~VideomtVideoProcessor.post_process_instance_segmentation`] or [`~VideomtVideoProcessor.post_process_panoptic_segmentation`] to compute final segmentation maps. Please, see [`~VideomtVideoProcessor`] for details regarding usage. ) custom_introc\rSrSr%SrSr\RS-\S'Sr \RS-\S'Sr \RS-\S'Sr \RS-\S'Sr \ \RS-\S'Sr\ \RS-\S 'S rg) %VideomtForUniversalSegmentationOutputi\a< loss (`torch.Tensor`, *optional*): The computed loss, returned when labels are present. class_queries_logits (`torch.FloatTensor`): A tensor of shape `(batch_size, num_queries, num_labels + 1)` representing the proposed classes for each query. Note the `+ 1` is needed because we incorporate the null class. masks_queries_logits (`torch.FloatTensor`): A tensor of shape `(batch_size, num_queries, height, width)` representing the proposed masks for each query. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Last hidden states (final feature map) of the last layer. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states all layers of the model. attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `tuple(torch.FloatTensor)` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Self and Cross Attentions weights from transformer decoder. Nlossclass_queries_logitsmasks_queries_logitslast_hidden_stater attentions)rErFrGrHrIr#rJ FloatTensor__annotations__r$r%r&rrr'rKr(r4r2r"r"\s&&*D%  d ")59%++d2959%++d2926u((4/659M5**+d2926Je''(4/6r4r"input_featurespoint_coordinatesc UR5S:XaSnURS5n[RRR "USU-S- 40UD6nU(aUR S5nU$)a A wrapper around `torch.nn.functional.grid_sample` to support 3D point_coordinates tensors. Args: input_features (`torch.Tensor` of shape (batch_size, channels, height, width)): A tensor that contains features map on a height * width grid point_coordinates (`torch.Tensor` of shape (batch_size, num_points, 2) or (batch_size, grid_height, grid_width,: 2)): A tensor that contains [0, 1] * [0, 1] normalized point coordinates add_dim (`bool`): boolean value to keep track of added dimension Returns: point_features (`torch.Tensor` of shape (batch_size, channels, num_points) or (batch_size, channels, height_grid, width_grid): A tensor that contains features for points in `point_coordinates`. rTr;g@?)rmrqrJrr grid_samplesqueeze)r+r,add_dimrpoint_featuress r2 sample_pointr3st(!#-77:XX((44^SK\E\_bEbmflmN'//2 r4inputslabelscUR5RS5nS[R"XR5-nUR S5SS2S4UR S5SSS24-nSUS-US-- - nU$)a A pair wise version of the dice loss, see `dice_loss` for usage. Args: inputs (`torch.Tensor`): A tensor representing a mask labels (`torch.Tensor`): A tensor with the same shape as inputs. Stores the binary classification labels for each element in inputs (0 for the negative class and 1 for the positive class). Returns: `torch.Tensor`: The computed loss between each pairs. rr;rTN)sigmoidr@rJrTsum)r4r5 numerator denominatorr#s r2pair_wise_dice_lossr<sz^^  % %a (FELL22I**R.D)FJJrN47,CCK A +/2 2D Kr4cZURSn[R"SS9nU"U[R"U55nU"U[R "U55n[R "XB- UR5n[R "XR- SU- R5nXg-nU$)a A pair wise version of the cross entropy loss, see `sigmoid_cross_entropy_loss` for usage. Args: inputs (`torch.Tensor`): A tensor representing a mask. labels (`torch.Tensor`): A tensor with the same shape as inputs. Stores the binary classification labels for each element in inputs (0 for the negative class and 1 for the positive class). Returns: loss (`torch.Tensor`): The computed loss between each pairs. rnone reduction)r<rBCEWithLogitsLossrJ ones_like zeros_likerr8) r4r5height_and_width criterioncross_entropy_loss_poscross_entropy_loss_negloss_posloss_negr#s r2$pair_wise_sigmoid_cross_entropy_lossrJs||A$$v6I&vuv/FG&vu/?/?/GH||2EvxxPH||2EF ~~VH  D Kr4c ^\rSrSrSrSS\S\S\S\4U4Sjjjr\R"5S\RS \RS \RS \RS \ \ \ 4 S j5r SrU=r$)VideomtHungarianMatcheriaaThis class computes an assignment between the labels and the predictions of the network. For efficiency reasons, the labels don't include the no_object. Because of this, in general, there are more predictions than labels. In this case, we do a 1-to-1 matching of the best predictions, while the others are un-matched (and thus treated as non-objects). cost_class cost_mask cost_dice num_pointsc>[TU]5 US:XaUS:XaUS:Xa [S5eX@lXlX lX0lg)aCreates the matcher Params: cost_class (`float`, *optional*, defaults to 1.0): Relative weight of the classification error in the matching cost. cost_mask (`float`, *optional*, defaults to 1.0): This is the relative weight of the focal loss of the binary mask in the matching cost. cost_dice (`float`, *optional*, defaults to 1.0): This is the relative weight of the dice loss of the binary mask in the matching cost. num_points (`int`, *optional*, defaults to 12544): No. of points to sample on which the mask loss will be calculated. The same set of K points are uniformly sampled for all prediction and ground truth masks to construct the cost matrix for bipartite matching. rzAll costs can't be 0N)r"r#r=rPrMrNrO)r/rMrNrOrPr1s r2r# VideomtHungarianMatcher.__init__sC"  ?yA~)q.34 4$$""r4r%r$ mask_labels class_labelsr6c /nURSn[U5GHnX'RS5nXn USS2XG4*n X7RU 5n U SS2S4n U SS2S4n [R "SUR SU RS9n U RU RSSS5n [XSS9RS5n U RU RSSS5n[XSS9RS5n [X5n[X5nURU-URU --URU--n[R "U[R""S 55n[R$"U[R""S 55n[R&"US5n[)UR+55nUR-U5 GM UVVs/sHLunn[R."U[R0S 9[R."U[R0S 94PMN nnnU$s snnf) a Params: masks_queries_logits (`torch.Tensor`): A tensor of dim `batch_size, num_queries, num_labels` with the classification logits. class_queries_logits (`torch.Tensor`): A tensor of dim `batch_size, num_queries, height, width` with the predicted masks. class_labels (`torch.Tensor`): A tensor of dim `num_target_boxes` (where num_target_boxes is the number of ground-truth objects in the target) containing the class labels. mask_labels (`torch.Tensor`): A tensor of dim `num_target_boxes, height, width` containing the target masks. Returns: matched_indices (`list[tuple[Tensor]]`): A list of size batch_size, containing tuples of (index_i, index_j) where: - index_i is the indices of the selected predictions (in order) - index_j is the indices of the corresponding selected labels (in order) For each batch element, it holds: len(index_i) = len(index_j) = min(num_queries, num_target_boxes). rrTNrr;rkF align_cornersg _Bg _r9)r<rangerr>rJrrPrkrepeatr3r0rJr<rNrMrOminimumtensormaximum nan_to_numrcpuappend as_tensorint64)r/r%r$rSrTindicesrti pred_probs pred_maskrM target_maskr,target_coordinatespred_coordinatesrNrO cost_matrixassigned_indicesjmatched_indicess r2rCVideomtHungarianMatcher.forwards8*,*//2 z"A-088r?r)rrAmeanr9)r4r5rrrEcross_entropy_lossr#s r2sigmoid_cross_entropy_lossry^sB$$v6I"62  " "1 % ) ) +i 7D Kr4c^\rSrSrS\S\\\44U4SjjrS\ \ \ S\ \ 4Sjr S\ \ S\ \ \ 44S jrS \ S \ \ S \ \R S\\\ 44S jrS\RS\ \RS \ \R S\ S\\\R44 SjrSrSrS\RS\R4SjrS\RS\ S\ S\S\R4 SjrS S\RS \RS\ \RS \ \RS\\\R4S-S\\\R44 SjjrS \RS\R2S\R4SjrSrU=r$)! VideomtLossirr0 weight_dictc>[TU]5 [US/5 URUlX lUR Ul[R"URS-5nUR US'URSU5 URUl URUl URUl [URUR UR"URS9Ulg)a  The Videomt Loss. The loss is computed very similar to DETR. The process happens in two steps: 1) we compute hungarian assignment between ground truth masks and the outputs of the model 2) we supervise each pair of matched ground-truth / prediction (supervise class and mask) Args: config (`VideomtConfig`): The configuration for Videomt model also containing loss calculation specific parameters. weight_dict (`dict[str, float]`): A dictionary of weights to be applied to the different losses. scipyrrT empty_weightroN)r"r#r num_labelsr|no_object_weighteos_coefrJrrctrain_num_pointsrPoversample_ratioimportance_sample_ratiorL class_weight dice_weight mask_weightmatcher)r/r0r|rr1s r2r#VideomtLoss.__init__ss $ * ++&// zz$//A"56 == R ^\:!11 & 7 7'-'E'E$.**((((   r4sizesr6cpUSnUSSH'n[U5HupE[X$U5X$'M M) U$)Nrr) enumeratemax)r/rmaxessublistindexitems r2 _max_by_axisVideomtLoss._max_by_axissCaQRyG(1 "5<6  2! r4tensorscTURUVs/sHn[UR5PM sn5n[U5/U-nUupVpxUSRn USR n [ R"XIU S9n [ R"XWU4[ RU S9n [XU 5Houp-nU SURS2SURS2SURS24RU5 SUSURS2SURS24'Mq X4$s snf)Nrrrr;F) rrqr<lenr:rkrJrYrrpzipcopy_)r/rr\max_size batch_shapertrrvrwr:rkpadded_tensors padding_masks padded_tensor padding_masks r2_pad_images_to_max_in_batch'VideomtLoss._pad_images_to_max_in_batchs'$$w%OwVd6<<&8w%OP7|nx/ '2$ v   ""M J#>ejjY_` 36wP]3^ /F< +FLLO+->v||A->@Q&,,q/@QQ R X XY_ `AFL*6<<?*,=fll1o,== >4_,,&PsD%r$rTrcc UnURupVn[R"URS9nUR U5n [ R "[X#5V VV s/sH un up{XPM sn nn 5n [ R"XV4UR[ RURS9n XU 'URSS5nU"X5nSU0nU$s sn nn f)a-Compute the losses related to the labels using cross entropy. Args: class_queries_logits (`torch.Tensor`): A tensor of shape `batch_size, num_queries, num_labels` class_labels (`list[torch.Tensor]`): List of class labels of shape `(labels)`. indices (`tuple[np.array])`: The indices computed by the Hungarian matcher. Returns: `dict[str, Tensor]`: A dict of `torch.Tensor` containing the following key: - **loss_cross_entropy** -- The loss computed using cross entropy on the predicted and ground truth labels. )r?) fill_valuer:rkrr;loss_cross_entropy) r<rCrossEntropyLossr$_get_predictions_permutation_indicesrJrsrfullrrbrkrA)r/r$rTrc pred_logitsrt num_queriesrrEidxtargetrltarget_classes_otarget_classespred_logits_transposedloss_celossess r2 loss_labelsVideomtLoss.loss_labelss"+ %0%6%6" ''t/@/@A 77@ 99-0-G H-G>66AVY-G H   %$//]h]o]o /s!,!6!6q!!<2C&0  Is C r%rSrrc^TRU5nTRU5nXnTRU5upXnUSS2S4nUSS2S4n[R"5 TR UU4SjTR TRTR5n [XSS9RS5n SSS5 [UW SS9RS5n [U W U5[XU5S.n AAU $!,(df  NF=f)a$Compute the losses related to the masks using sigmoid_cross_entropy_loss and dice loss. Args: masks_queries_logits (`torch.Tensor`): A tensor of shape `(batch_size, num_queries, height, width)`. mask_labels (`torch.Tensor`): List of mask labels of shape `(labels, height, width)`. indices (`tuple[np.array])`: The indices computed by the Hungarian matcher. num_masks (`int)`: The number of masks, used for normalization. Returns: losses (`dict[str, Tensor]`): A dict of `torch.Tensor` containing two keys: - **loss_mask** -- The loss computed using sigmoid cross entropy loss on the predicted and ground truth. masks. - **loss_dice** -- The loss computed using dice loss on the predicted on the predicted and ground truth, masks. Nc&>TRU5$r{)calculate_uncertainty)logitsr/s r2(VideomtLoss.loss_masks..st99&Ar4FrWr) loss_mask loss_dice) r _get_targets_permutation_indicesrrJrpsample_points_using_uncertaintyrPrrr3r0ryru)r/r%rSrcrrsrc_idxtgt_idx pred_masks target_masksrr, point_labels point_logitsrs` r2 loss_masksVideomtLoss.loss_maskss4;;GD77@)2 ::;G #,  4( #AtG, ]]_ $ D DA%%,, ! ( W\]eefghL$J0AQVW__`ab 4L,PYZ"?"JJyaPVP]P]^#FUS 2<@"#:#GH&=jj,Q1W59MSTUVWX"U\\)5::V\VcVc%dd QW~-222q9#((2,/JOOPYqrs q % "EJJyQW]WdWd$ef! ! r4Nauxiliary_predictionscURXX45nURXDSRS9n0URXXg5EUR X$U5EnUbk[ U5H\upU SnU SnUR XX45n U R5V V s0sH upU SU 3U _M n n n URU 5 M^ U$s sn n f)a This performs the loss computation. Args: masks_queries_logits (`torch.Tensor`): A tensor of shape `(batch_size, num_queries, height, width)`. class_queries_logits (`torch.Tensor`): A tensor of shape `(batch_size, num_queries, num_labels)`. mask_labels (`torch.Tensor`): List of mask labels of shape `(labels, height, width)`. class_labels (`list[torch.Tensor]`): List of class labels of shape `(labels)`. auxiliary_predictions (`dict[str, torch.Tensor]`, *optional*): if `use_auxiliary_loss` was set to `true` in [`VideomtConfig`], then it contains the logits from the inner layers of the VideomtMaskedAttentionDecoder. Returns: losses (`dict[str, Tensor]`): A dict of `torch.Tensor` containing three keys: - **loss_cross_entropy** -- The loss computed using cross entropy on the predicted and ground truth labels. - **loss_mask** -- The loss computed using sigmoid cross_entropy loss on the predicted and ground truth masks. - **loss_dice** -- The loss computed using dice loss on the predicted on the predicted and ground truth masks. if `use_auxiliary_loss` was set to `true` in [`VideomtConfig`], the dictionary contains additional losses for each auxiliary predictions. rrVr%r$r) r get_num_masksrkrrrrCitemsupdate)r/r%r$rSrTrrcrrrr aux_outputs loss_dictrrs r2rCVideomtLoss.forwardcsH,,3;e&&|O,VideomtLoss.get_num_masks..sALG Lsrr)min) r9rJrarrr _shared_stater num_processesclamp)r/rTrkrr world_sizes r2rVideomtLoss.get_num_maskssvALAA OOIU[[P  " $ $))R/"9- )^99 KK 6A> r4)rrrrrPrr|r{)rErFrGrHrdictrrr#rqrrrrrnparrayrrJrrrrrrCrkrrKrLrMs@r2r{r{rs ! }! 4U ;K! F$tCy/d3i-4<-E&RX.DY-" $* :>v, QVWYW_W_Q`  c6k  D<#ll<%,,'<rxx <  < c5<< <|2 - "ELL"U\\""5! 5! 5!  5! "' 5! 5!zAE 5#ll5$ll5%,,' 5 5<<( 5 $C$56= 5 c5<< 5n %,,  QVQ]Q]  r4r{c\rSrSr%Sr\\S'SrSrSr Sr S/r S r \ \S .r\R""5S \R&S S 4Sj5rSrg )VideomtPreTrainedModelizz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. r0videomtpixel_values_videos)videoFrT)rr'rr6NcURRn[U[R[R [R 45(a[R"UR[R"S5S9 URby[RRRUR5up4US:aS[R"U5- OSn[R"URU*U5 GO[U[R 5(aB[R""UR5 [R$"UR5 GOC[U[R&5(ax[R("URSSS9 UR*bI[-URSS5(d-[R$"URUR*5 GO[U[.5(aH[1US 5(a5[R2"UR4URR65 GOO[U[85(a[R:"UR<SUS9 [R$"UR>5 [R@"URB[RD"URBRFS 5RIS 55 O[U[J5(aT[RL"URNS-5nURPUS '[R@"URRU5 O5[U[T5(a [R""URV5 [U[85(a*[RR%URX5 gg) Nrj)arrr)rwstd_is_hf_initializedFrrTrS)-r0initializer_ranger(rrr-ConvTranspose2dinitkaiming_uniform_r?mathsqrtrrJ_calculate_fan_in_and_fan_outuniform_rones_zeros_ranormal_ padding_idxgetattrrhasattr constant_rrrO trunc_normal_rXr[rrRrdr<rer{rrrrVideomtForUniversalSegmentationattn_mask_probsrf)r/rrfan_inrboundrs r2 _init_weights$VideomtPreTrainedModel._init_weightsswkk++ fryy"))R5G5GH I I  ! !&--499Q< @{{&!HHMMGG V 17!DIIf-- fkkE659  - - JJv}} % KK $  - - LLSa 8!!-gfmmMach6i6i FMM&*<*<=>  1 2 2vy))v~~t{{/K/KL  1 2 2   v//cs C KK.. / JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h  , , ::f&7&7!&;[TU]XUS9 g)N)relementwise_affine)r"r#)r/r&raffiner1s r2r#VideomtLayerNorm2d.__init__s 6Jr4rr6cURSSSS5n[R"XRURUR UR 5nURSSSS5nU$)Nrr;rr)permuteF layer_normnormalized_shaper?rrrs r2rCVideomtLayerNorm2d.forwards`#++Aq!Q7 ||L2G2GVZV_V_aeaiaij #++Aq!Q7 r4r()gư>TrrMs@r2r r s)KELLU\\r4r cj^\rSrSrS\4U4SjjrS\RS\R4SjrSr U=r $)VideomtScaleLayerir0c >[TU]5 URn[R"X"SSS9Ul[ URUl[R"UUSSUSS9Ul [U5Ul g)Nr;rrrF)r paddinggroupsr) r"r#r'rrconv1r rrr-conv2r  layernorm2dr/r0r'r1s r2r#VideomtScaleLayer.__init__ss (( '' aXYZ  !2!23YY     .k:r4rr6cURU5nURU5nURU5nURU5nU$r{)r0rr1r2rs r2rCVideomtScaleLayer.forwardsB =1  6  =1 ((7 r4)rr0r1r2 rErFrGrHrr#rJrrCrKrLrMs@r2r,r,s/;}; U\\ellr4r,cj^\rSrSrS\4U4SjjrS\RS\R4SjrSr U=r $)VideomtScaleBlockir0c>[TU]5 URUl[R "[ UR5Vs/sHn[U5PM sn5Ulgs snfr{) r"r#num_upscale_blocks num_blocksr ModuleListrYr,blockr/r0rr1s r2r#VideomtScaleBlock.__init__sN  33]]uT__G]#^G]!$5f$=G]#^_ #^sA*rr6c<URH nU"U5nM U$r{)r>)r/rr>s r2rCVideomtScaleBlock.forward s ZZE!-0M r4)r>r<r7rMs@r2r9r9s1`}` U\\ellr4r9cj^\rSrSrS\4U4SjjrS\RS\R4SjrSr U=r $)VideomtMaskHeadir0c >[TU]5 URn[R"X"5Ul[R"X"5Ul[R"X"5Ul[URUl gr{) r"r#r'rrrrfc3r rrr3s r2r#VideomtMaskHead.__init__s[ (( 99[699[699[6 !2!23r4rr6cURURU55nURURU55nURU5nU$r{rrrrFrs r2rCVideomtMaskHead.forwardsD(?@ (?@ / r4rIr7rMs@r2rDrDs/4}4U\\ellr4rDzY The Videomt Model with head on top for instance/semantic/panoptic segmentation. cz^\rSrSrSrS\4U4SjjrS\S\S\S\S \\ \4S \\ \44 S jr S \\ \4S \4S jr \ \ \SS\RS-S\\RS-S\\RS-S\\RS-S\\S \4 Sjj555rSrS\R4SjrSrU=r$)ri rr0c>>[TU]U5 XlURUl[ U5Ul[ R"URURS9Ul [ R"URUR5Ul [ R"[UR5Vs/sHn[!U5PM sn5Ul[%U5Ul[)U5Ul[ R,"URUR.S-5UlUR2UR4-UR2UR4-4UlUR8UR:UR<S.Ul[AXR>S9Ul!URES[FRH"URJ55 [ R,"URUR5Ul&URO5 gs snf)Nrr)rrr)r0r|r)(r"r#r0num_hidden_layersrOrBrrr'r layernormrarrr=rYrlayersr9 upscale_blockrD mask_headrrclass_predictorr$r% grid_sizerrrr|r{rErcrJrr< query_updater post_initr?s r2r#(VideomtForUniversalSegmentation.__init__(s   !'!9!9+F3f&8&8f>S>ST\\&"4"4f6H6HI mm5IaIaCb$cCba\&%9Cb$cd .v6(0!yy););V=N=NQR=RS ++v/@/@@&BSBSW]WhWhBhi"("5"5++++.  %F@P@PQ . 6;L;L0MNYYv'9'96;M;MN '%ds=Hr%r$rSrTrr6cURUUUUUS9nURR5H)upxUR5HupXy;dM X-n M M+ U$)N)r%r$rSrTr)rEr|r) r/r%r$rSrTrrrr?loss_keyr#s r2 get_loss_dict-VideomtForUniversalSegmentation.get_loss_dictEsj(,~~!5!5#%"7 (6(  ++113KC"+//"3?ND#44 r4rc4[UR55$r{)r9r)r/rs r2get_loss(VideomtForUniversalSegmentation.get_loss]s9##%&&r4N patch_offsetsrc vSU;a [S5eUc [S5eURS:wa [S5eUcUb [S5eURupgpn URXg-XU 5n UR U 5n UR UR R- n URSU H nU"U 5n M U RXgU RSU RS 5n /n/n/nSn[U5GHnnU SS2U4nUcKURRSSS2SS24RUS S 5RUR5nOdUR!U5RUR5URRSSS2SS24RUR5-n["R$"UU4SS 9nURU SH nU"U5nM UR'U5nUR)U5unnUR+U5 UR+U5 UR+U5 USS2SUR R,2SS24nGMq [/S["R$"US S 9["R$"US S 9["R$"US S 9S 9$)a pixel_values_videos (`torch.Tensor`, *optional*): Video inputs of shape `(batch_size, num_frames, num_channels, height, width)`. mask_labels (`list[torch.Tensor]`, *optional*): Not supported for 5D video inputs. class_labels (`list[torch.LongTensor]`, *optional*): Not supported for 5D video inputs. patch_offsets (`list[torch.Tensor]`, *optional*): Unused for video inputs and only kept for modular compatibility. r5zAUse `pixel_values_videos` with `VideomtForUniversalSegmentation`.Nz'You have to specify pixel_values_videosrjzyVideomtForUniversalSegmentation only supports 5D video inputs of shape (batch_size, num_frames, channels, height, width).zTraining with 5D video inputs is not supported in `VideomtForUniversalSegmentation`. Flatten frames and use `EomtForUniversalSegmentation` instead.rr;rTrlr)r#r%r$r&)r=rnr<rorBrMr0r<rOrrYrr?rer>rkrTrJrsrNpredictr`rr")r/rrSrTr^rrtrur&rvrwflat_pixel_valuesrquery_start_idx layer_moduleall_masks_queries_logitsall_class_queries_logitsall_last_hidden_statespropagated_query frame_idxframe_hidden_states query_tokenssequence_outputr%r$s r2rC'VideomtForUniversalSegmentation.forward`s* V #`a a  &FG G  # #q (E   "l&>Q  ?R>W>W; e/77 8OQ]glm(9: 004;;3I3II KK(89L(7M:&**:=CVCVWXCY[h[n[nop[qr #% #% !#z*I"/9 "= '#zz00q!<CCJPRTVWZZ[n[u[uv #112BCFFGZGaGabeieoeoevev!QJf"(//0 1 #())\;N,OUV"W  $ O,< = &23F&G#!>#nn-@AO9=o9V 6 "6 $ + +,@ A $ + +,@ A " ) )/ :216O 8O8O6OQR3RS )+,5!&+C!K!&+C!K#ii(>AF   r4c.URR$r{)rBr\rs r2get_input_embeddings4VideomtForUniversalSegmentation.get_input_embeddingss///r4rcUSS2SURR2SS24nURU5nUSS2URRURR-S2SS24nUR SS5nUR "URSS/URQ76nURU5nURU5n[R"SX$5nXS4$)Nrr;rrTzbqc, bchw -> bqhw) r0rrRrBr`rAror<rSrQrPrJeinsum)r/rrj class_logits prefix_tokens mask_logitss r2r`'VideomtForUniversalSegmentation.predictsa!:4;;#:#:!:A=> ++L9 q$++"9"9DOO<]<]"]"_abbc %//15 %--m.A.A!.DbZ4>>Z ~~l3 **=9 ll#6 T ((r4) rRr0rErBrSrNrOrQrMrrTrPr|)NNNN)rErFrGrHrrr#rrrrYr\rrrrJrqrrr"rCrnr`rKrLrMs@r2rr sK ,O}:$%    $CK0  c6k 0'$sF{"3'' 48152637 O "\\D0O %,,'$.O 5<<(4/ O ELL)D0 O +, O  /O  O b0)ell))r4r)r)rF)F)Mcollections.abcr)rr dataclassesrnumpyrrJtorch.nn.functionalrrr'rrr activationsr file_utilsr r r modeling_layersr modeling_utilsrrprocessing_utilsrutilsrrr utils.genericrutils.output_capturingrconfiguration_videomtrscipy.optimizer accelerateraccelerate.utilsrrrrOr~rrrrrprrrrrr"r3r<rJrLrruryr{rrr r,r9rDr__all__r(r4r2rs1* $! &!LL9F&PP7504'' RYY F, ,^&(bii(0% II% <<% % << % LL4' %  %%.8)ryy8)v U\\ e T V[VbVb  %bii %(ryy("'-'T+ +  7K7  7<LQLL5:\\ \\@6,u||X]XdXd8gbiigTff<u||U\\VY^c^j^j(u"))up 1._1.1.h 2  bii" `)&<`)  `)F $%F Gr4