Zj$SrSSKrSSKJr SSKrSSKJr SSKJrJrJ r SSK J r SSK Jr SS KJrJrJr SS KJr SS KJrJr SS KJr SS KJr SSKJrJrJrJ r J!r!J"r"J#r# SSK$J%r%J&r& SSK'J(r( SSK)J*r*J+r+J,r,J-r-J.r.J/r/J0r0 SSK1J2r2 SSK3J4r4J5r5 SSK6J7r7 \-"5(a\/Rp"\95r:S\RvS\<4Sjr="SS\R|5r?"SS\R|5r@SDS\RS\RvS\RvS \RvS!\RvS-S"\BS-S#\BS$\(\*4S%jjrC"S&S'\R5rD"S(S)\5rE"S*S+\5rF"S,S-\R5rG\+"S.S/\&55rH"S0S1\H5rI"S2S3\H5rJ\+"S4S5\H55rK\+"S6S79"S8S9\H\55rL\+"S:S79"S;S<\H55rM\+"S=S>\H55rN"S?S@\H5rO"SASB\H\5rP/SCQrQg)EzPyTorch MBART model.N)Callable)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleis_torch_flex_attn_availableis_torchdynamo_compilingloggingtorch_compilable_check)merge_with_config_defaults)OutputRecordercapture_outputs) MBartConfig input_ids pad_token_idc`UR5nUc [S5eURUS:HU5 URU5R SS9S- R S5nUR SU5R5nUSS2SS24R5USS2SS24'XBSS2S4'U$)z Shift input ids one token to the right, and wrap the last non pad token (the token) Note that MBart does not have a single `decoder_start_token_id` in contrast to other Bart-like models. Nz1self.model.config.pad_token_id has to be defined.ir'dimr)clone ValueError masked_fill_nesum unsqueezegathersqueeze)r)r*prev_output_tokens index_of_eosdecoder_start_tokenss y/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/mbart/modeling_mbart.pyshift_tokens_rightr;@s #*LMM##$6$$> M&)),7;;;BQFQQRTUL-44Q EMMO 21crc6 : @ @ Bq!"u3q!t c^\rSrSrSrS\S\4U4SjjrS S\RS\S \RS-4U4S jjjr S r U=r $) MBartLearnedPositionalEmbeddingUzF This module learns positional embeddings up to a fixed maximum size. num_embeddings embedding_dimcL>SUl[TU] XR-U5 gN)offsetsuper__init__)selfr@rA __class__s r:rG(MBartLearnedPositionalEmbedding.__init__Zs"  ++5}Er<Nr)past_key_values_length position_idsc>Uc]URSSupE[R"X"U-[RURR S9R US5nOURS5n[TU]%X0R-5$)z3`input_ids' shape is expected to be [bsz x seqlen].NrD)dtypedevicer.r) shapetorcharangelongweightrOexpandr4rFforwardrE)rHr)rKrLbszseq_lenrIs r:rV'MBartLearnedPositionalEmbedding.forward`s  $??2A.LC <<&(HPUPZPZcgcncncucufS"o (11!4Lw|kk9::r<)rE)rN) __name__ __module__ __qualname____firstlineno____doc__intrGrQTensorrV__static_attributes__ __classcell__rIs@r:r>r>UsWFsF3Fmq ; ;?B ;V[VbVbeiVi ; ;r<r>c r^\rSrSrSrS S\S\S\S\S-4U4SjjjrS \R4U4S jjr S r U=r $) MBartScaledWordEmbeddingqzT This module overrides nn.Embeddings' forward by multiplying with embeddings scale. r@rA padding_idx embed_scaleNc2>[TU]XU5 X@lgN)rFrGrh)rHr@rArgrhrIs r:rG!MBartScaledWordEmbedding.__init__vs  D&r<r)c<>[TU]U5UR-$rj)rFrVrh)rHr)rIs r:rV MBartScaledWordEmbedding.forwardzswy)D,<,<<L==((6??([L,,|3K''1-88:K  $$r<c$^\rSrSrSrSS\S\S\S\S\S \S \S-S \S-4U4S jjjr SS \ RS\ RS-S\ S-S\ RS-S\ \S\\ R\ RS-44 SjjrSrU=r$)MBartAttentionz=Multi-headed attention from 'Attention Is All You Need' paperN embed_dim num_headsrx is_decoderbias is_causalconfig layer_idxc t>[T U]5 XlX lX0lX-UlXplUR U-UR:wa[SURSUS35eUR S-UlX@l X`l Xl Uc>UR(a-[RSURRS35 [ R""XUS9Ul[ R""XUS9Ul[ R""XUS9Ul[ R""XUS9Ulg)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r{zInstantiating a decoder z without passing `layer_idx` is not recommended and will lead to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.r)rFrGrrrxhead_dimrr0rwrrrlogger warning_oncerIrZrLineark_projv_projq_projout_proj) rHrrrxrrrrrrIs r:rGMBartAttention.__init__s  "" !.  MMI %$.. 8MdnnM]$YKr3 }}d* $""     *4>>+B+B*CD,,  ii 4@ ii 4@ ii 4@  )TB r< hidden_stateskey_value_statespast_key_valuesrvryreturnc ,USLnURSSn/UQSPURP7nURU5RU5R SS5n Sn Ub][ U[ 5(aFURRUR5n U(a URn OURn OUn U(aUOUn U(aQUbNU (aGW RURRn U RURRnOURU 5n UR!U 5n/U RSSQSPURP7nU RU5R SS5n URU5R SS5nUbSW R#XUR5upU(a.[ U[ 5(aSURUR'[$R&"UR(R*[,5nU"UU U UU4UR.(dSO UR0UR2S.UD6unnUR4"/UQSP76R75nUR9U5nUU4$) z#Input shape: Batch x Time x ChannelNr.r'rDFT)rxrw)rPrrviewr isinstancer is_updatedgetrcross_attention_cacheself_attention_cachelayerskeysvaluesrrupdater get_interfacer_attn_implementationrr~rxrwreshaperr)rHrrrrvryis_cross_attention input_shape hidden_shape query_statesrcurr_past_key_valuescurrent_states key_states value_stateskv_shapeattention_interfacerrs r:rVMBartAttention.forwardsc.T9$))#2. 88b8$--8 {{=166|DNNqRST   &/+>??,77;;DNNK %+:+P+P(+:+O+O('6$-?)] /"=*-44T^^DIIJ/66t~~FMML^4J;;~6LF--cr2FBF FH#2<(?(M(M KK , ,.E) %8      % $}}C$,,LL %  % ! \"));;;;FFH mmK0 L((r<) rrxrrrrrrrrrrwr)rFTFNNNNN)rZr[r\r]r^r_rqboolr(rGrQr`r rrtuplerVrarbrcs@r:rrsG  %) $%C%C%C %C  %C  %C%Cd"%C:%C%CT15(,.2 H)||H) ,,-H) H)  t+ H)-.H) u||U\\D00 1H)H)r<rc^\rSrSrS\4U4SjjrS\RS\RS\\ S\R4Sjr S r U=r $) MBartEncoderLayerircj>[TU]5 URUl[ URUR UR US9Ul[R"UR5Ul URUl [URUlURUl[R "URUR"5Ul[R "UR"UR5Ul[R"UR5Ulg)N)rrrxr)rFrGd_modelrrencoder_attention_headsattention_dropout self_attnr LayerNormself_attn_layer_normrxr activation_function activation_fnactivation_dropoutrencoder_ffn_dimfc1fc2final_layer_normrHrrIs r:rGMBartEncoderLayer.__init__s 'nn44,,   %'LL$@!~~ #F$>$>?"(";";99T^^V-C-CD99V33T^^D " T^^ <r<rrvryrc UnURU5nUR"SUUS.UD6up[RR XRUR S9nXA-nUnUR U5nURURU55n[RR XRUR S9nURU5n[RR XRUR S9nXA-nUR[R:XaC[R"UR5RS- n[R "X*US9nU$)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. )rrvr|i)minmax)rrrrrxr~rrrrrrNrQfloat16finforclamp)rHrrvryresidual_ clamp_values r:rVMBartEncoderLayer.forward$sD!11-@ >> ')    --m||VZVcVc-d  0  --m< **488M+BC  --m?V?Vaeanan-o /  --m||VZVcVc-d  0   %-- /++m&9&9:>>EK!KK <[YMr<) rrrxrrrrrr) rZr[r\r]r(rGrQr`rrrVrarbrcs@r:rrsQ={=$"||" "+, "  ""r<rc^\rSrSrSS\S\S-4U4SjjjrSS\RS\RS-S\RS-S \RS-S \ S-S \ S-S \ \ S \R4Sjjr SrU=r$)MBartDecoderLayeriINrrc >[TU]5 URUl[ URUR UR SSUUS9UlURUl[URUl URUl [R"UR5Ul[ URUR UR SUUS9Ul[R"UR5Ul[R$"URUR&5Ul[R$"UR&UR5Ul[R"UR5Ulg)NT)rrrxrrrr)rxrrr)rFrGrrrdecoder_attention_headsrrrxr rrrrrr encoder_attnencoder_attn_layer_normrdecoder_ffn_dimrrr)rHrrrIs r:rGMBartDecoderLayer.__init__Js  'nn44,, ~~ #F$>$>?"(";";$&LL$@!* NN  * *,,  (*||DNN'C$99T^^V-C-CD99V33T^^D " T^^ <r<rrvencoder_hidden_statesencoder_attention_maskr use_cacheryrc UnURU5nUR"SUUUS.UD6up[RR XRUR S9nX-nUbbUnUR U5nUR"SUUUUS.UD6up[RR XRUR S9nX-nUnURU5nURURU55n[RR XRUR S9nURU5n[RR XRUR S9nX-nU$)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. past_key_values (`Cache`): cached past key and value projection states )rrrvr|)rrrvrr) rrrrrxr~rrrrrrr) rHrrvrrrrryrrs r:rVMBartDecoderLayer.forwardis*!11-@  >> '+)     --m||VZVcVc-d  0  ! ,$H 88GM#00 +!65 /      MMM11-<[TU]5 [R"X5Ul[R "US9Ul[R"X#5Ulg)N)r})rFrGrrdenseDropoutrxr)rHrrrrrIs r:rG MBartClassificationHead.__init__s@ YYy4 zzN3  )9 r<rrcURU5nURU5n[R"U5nURU5nUR U5nU$rj)rxrrQtanhr)rHrs r:rVMBartClassificationHead.forwardsN ]3  =1  =1  ]3  m4 r<)rrxrrprcs@r:rrsQ7 : : : :  :U\\ellr<rcd^\rSrSr%\\S'SrSr/SQrSr Sr Sr Sr U4Sjr \S5rSrU=r$) MBartPreTrainedModelirmodelT)rrrc>[TU]U5 [U[5(a![R "UR 5 ggrj)rF _init_weightsrMBartForConditionalGenerationinitzeros_final_logits_bias)rHrrrIs r:r"MBartPreTrainedModel._init_weightss5 f% f; < < KK00 1 =r<cURRn[R"/SQSSSSU//URS9nUR U5US.nU$)N)r rDr rDrO)rvr))rr*rQtensorrOr2)rH pad_tokenr) dummy_inputss r:r!MBartPreTrainedModel.dummy_inputssWKK,, LL"2Q2q)4L!MVZVaVab 'll95" r<r)rZr[r\r]r(__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraphrpropertyrrarbrcs@r:rrsK &*#TN!2 r<rc^\rSrSrSr\\"\SSS9S.rS\ 4U4Sjjr S r \ \ SS \RS -S \R S -S \R"S -S\\S\\-4 Sjj55rSrU=r$) MBartEncoderiz Transformer encoder consisting of *config.encoder_layers* self attention layers. Each layer is a [`MBartEncoderLayer`]. Args: config: MBartConfig embed_tokens (nn.Embedding): output embedding r'rindex layer_name)r attentionsrc>[TU]U5 URUlURUlUR nUR UlURUl UR(a[R"U5OSn[URX RUS9Ul[!URU5Ul[$R&"[)UR*5Vs/sHn[-U5PM sn5UlXl[$R2"U5Ul[$R2"UR 5UlSUlUR;5 gs snf)NrornF)rFrGrxencoder_layerdrop layerdroprr*rgmax_position_embeddingsmax_source_positionsscale_embeddingmathsqrtre vocab_size embed_tokensr>embed_positionsr ModuleListrangeencoder_layersrrrrlayernorm_embedding layer_normgradient_checkpointing post_init)rHrrrhrrIs r:rGMBartEncoder.__init__s  ~~ 11NN !..$*$B$B!.4.D.Ddii *# 4   y*:*:   ?  * *  mmfNcNcHd$eHd1%6v%>Hd$ef  #%<< #: ,,v~~6&+# %fs0E/cUR(a.[URSS5(aUR5 ggg)Nr)F)r getattrrgradient_checkpointing_enablerHs r:._backward_compatibility_gradient_checkpointing;MBartEncoder._backward_compatibility_gradient_checkpointings4  / /GDKKIach4i4i  . . 05j /r<Nr)rv inputs_embedsryrc xUSLUSL- (a [S5eUcURU5nURUS5nX5RUR5-nUR U5n[ RRX`RURS9n[URUUS9n[UR5HRupxSn UR(a'[R"/5n XR :aSn U (aMHU"UU40UD6nMT UR#U5n[%US9$) a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. Nz:You must specify exactly one of input_ids or inputs_embeds).r.r|)rr2rvFT)last_hidden_state)r0r"r#torOr'rrrxr~rr enumeraterrQrandrr(r) rHr)rvr2ry embed_posridx encoder_layerto_dropdropout_probabilitys r:rVMBartEncoder.forward s*>  -t"; <YZ Z   --i8M((w)?@ % ]5I5I(JJ 00?  --m||VZVcVc-d 2;;')  #,DKK"8 CG}}&+jjn#&7"G7 -!"!! #9 6 ??r<) rrxr#r"r)r(rr'rrrgr)rZr[r\r]r^rr%r_can_record_outputsr(rGr0r$r&rQ LongTensorr` FloatTensorrrrrrVrarbrcs@r:rrs+$^1U {81  .2.226 @@##d*@@ t+@@((4/ @@ +, @@  @@ @@r<rcF^\rSrSrSr\\"\SSS9\"\SSS9S.rS\ 4U4S jjr \ \ SS \ RS -S \ RS -S \ R S -S\ RS -S\S -S\ R S -S\S -S\\S\\-4Sjj55rSrU=r$) MBartDecoderiOz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`MBartDecoderLayer`] Args: config: MBartConfig embed_tokens (nn.Embedding): output embedding r'rrr)rrcross_attentionsrc >[TU]U5 URUlURUlUR UlURUlUR(a [R"UR5OSn[URURUR US9Ul[!URUR5Ul[$R&"[)UR*5Vs/sH n[-XS9PM sn5UlXl[$R2"UR5Ul[$R2"UR5UlSUlUR;5 gs snf)Nrorn)rF)rFrGrxdecoder_layerdroprr*rgrmax_target_positionsrrr rrer!r"r>r#rr$r%decoder_layersrrrrr'r(r)r*)rHrrhirIs r:rGMBartDecoder.__init__^s"  ~~ 11!..$*$B$B!393I3Idii/s 4   v~~t/?/?[  ?  * * NN mmUZ[a[p[pUq$rUqPQ%6v%KUq$rs  #%<<#? ,,v~~6&+# %ssF Nr)rvrrrr2rryrc USLUSL- (a [S5eUcURU5nU(abUc_UcURR(a.[ [ URS9[ URS95O[ URS9nUR 5SSupUbUR5OSn [R"XRS9U -n Uc2[5(d#X-n [R"XURS9n[U[5(a UROUn[URUUUS9n[!URUUUS9nUR#[$XS 9n XlR'UR5-nUR)U5n[*R,R/UUR.UR0S 9n[3UR45HNunnUR0(a([R6"/5nUUR8:aM?U"UUU4UUUS .UD6nMP UR;U5n[=UUS 9$) a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same time)rr.rr)rr2rvr)rr2rvr)rLr|)rrr)r4r)r0r"ris_encoder_decoderr r rget_seq_lengthrQrRrOr!onesrrrrr#inputr5r'rrrxr~r6rr7rr(r)rHr)rvrrrr2rry batch_size seq_lengthrKrLmask_seq_lengthself_attn_cache causal_maskrr9 decoder_layerr<s r:rVMBartDecoder.forwardxsNn  -t"; <st t   --i8M 0)4 8V8V$L $DlZ^ZeZeFfg!5  "/!3!3!5cr!: ETE`!?!?!Afg||J7K7KLOee  !*B*D*D4AO"ZZ ML`L`aN/+>??  0 0   );;')+  ";;;'1"7 " ++E3I+e % 8L8L(MM 00?  --mt||VZVcVc-d "+DKK"8 C}}&+jjn#&7)%(> /# M#9" 6 8++  r<) rrxr#r"r)r(rr'rrFrg)NNNNNNN)rZr[r\r]r^rr%rr>r(rGr$r&rQr?r`r@r rrrrrrVrarbrcs@r:rBrBOs+$^1U*>~^ {4 .2.2:>:>(,26!%} ##d*}  t+}  %0047 } !& 0 04 7 }  } ((4/} $;} +,}  : :}  } r<rBc^\rSrSrSSS.rS\4U4SjjrSrSr\ \ SS \ RS-S \ RS-S \ RS-S \ RS-S \\\ RS-S\S-S\ RS-S\ RS-S\S-S\S-S\\S\\\ R-4Sjj55rSrU=r$) MBartModeliz shared.weight)zdecoder.embed_tokens.weightzencoder.embed_tokens.weightrcJ>[TU]U5 URURp2UR(a [ R "UR5OSn[X1RX$S9Ul [U5Ul [U5Ul UR5 g)Nrorn)rFrGr*r!rrr rresharedrencoderrBdecoderr*)rHrrgr!rhrIs r:rGMBartModel.__init__su  "("5"5v7H7HZ393I3Idii/s .z>>;p #F+ #F+  r<cUR$rj)rYr/s r:get_input_embeddingsMBartModel.get_input_embeddingss {{r<c|XlURURlURURlgrj)rYrZr"r[rHrus r:set_input_embeddingsMBartModel.set_input_embeddingss' $(KK !$(KK !r<Nr)rvdecoder_input_idsdecoder_attention_maskencoder_outputsrr2decoder_inputs_embedsr return_dictryrc nU bU OURRn Uc"Uc[XRR5nUcUR"S UUUU S.U D6nORU (aK[ U[ 5(d6[ US[U5S:aUSOS[U5S:aUSOSS9nUR"S UUUSUUUU U S.U D6n U (dX-$[U RU RU RU RU RURURURS9$) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) MBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 25004 for *en_XX*, and 25003 for *de_DE*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. N)r)rvr2rhrr'rD)r4rr)r)rvrrrr2rrh)r4rdecoder_hidden_statesdecoder_attentionsrCencoder_last_hidden_staterencoder_attentionsr)ruse_return_dictr;r*rZrrlenr[rr4rrrrC) rHr)rvrdrerfrr2rgrrhrydecoder_outputss r:rVMBartModel.forwards\J&1%)F 29kk>V>V W   ""ll#-+'   OO_!M!M-"1!"4474H14Loa0RV14_1E1I?1-tO,,  '1"1!"4#1+/#    "4 4!-??+;;"1"?"?.99,==&5&G&G"1"?"?.99  r<)r[rZrY NNNNNNNNNN)rZr[r\r]_tied_weights_keysr(rGr^rbrrrQr?r`rr@r rrrrrVrarbrcs@r:rWrWs`(7'6 { 0 .2.259:>BF(,26:>!%#'S ##d*S  t+S !++d2 S !& 0 04 7 S uU%6%6784? S S ((4/S  %0047S $;S D[S +,S  eE$5$56 6S S r<rWz The MBART Model with a language modeling head. Can be used for summarization, after fine-tuning the pretrained models. ) custom_introc,^\rSrSrSrS/rSS0rS\4U4SjjrSS \ S \ S-S \ S \ R4U4S jjjr S \ S S4Sjr\SS\R"S-S\R$S-S\R"S-S\R"S-S\\\R(S-S\S-S\R(S-S\R(S-S\R"S-S\ S-S\ S-S\\S \\\R(-4Sjj5rS\R$4SjrSrU=r$) rinrrlm_head.weightzmodel.shared.weightrcv>[TU]U5 [U5UlUR S[ R "SURRR455 [R"URURRRSS9Ul UR5 g)Nrr'Fr)rFrGrWrregister_bufferrQzerosrYr@rrrlm_headr*rs r:rG&MBartForConditionalGeneration.__init__xs  '  0%++q$**BSBSBbBb>c2deyy1B1B1Q1QX]^  r<Nnew_num_tokenspad_to_multiple_of mean_resizingrcx>[TU]XU5nURURRS5 U$)Nr)rFresize_token_embeddings_resize_final_logits_biasrTrP)rHr|r}r~new_embeddingsrIs r:r5MBartForConditionalGeneration.resize_token_embeddingss<8]jk &&~'<'<'B'B1'EFr<c,URRSnX::aURSS2SU24nON[R"SX- 4URRS9n[R "URU/SS9nUR SU5 g)Nr.r'rr,r)rrPrQryrOcatrx)rHr|old_num_tokensnew_bias extra_biass r:r7MBartForConditionalGeneration._resize_final_logits_biass//55b9  +--a..@AHa)H%IRVRhRhRoRopJyy$"8"8*!E1MH 0(;r<r)rvrdrerfrr2rglabelsrrhryc U bU OURRn U bCU (a[RS5 Sn Uc"Uc[ XRR 5nUR "U4UUUUUUUU U S. U D6n URU S5UR-nSnU bF[5nU"URSURR5U RS55nU (dU4U SS-nUbU4U-$U$[UUU RU RU RU R U R"U R$U R&S9 $) u decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) MBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 25004 for *en_XX*, and 25003 for *de_DE*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example Translation: ```python >>> from transformers import AutoTokenizer, MBartForConditionalGeneration >>> model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-en-ro") >>> tokenizer = AutoTokenizer.from_pretrained("facebook/mbart-large-en-ro") >>> example_english_phrase = "42 is the answer" >>> inputs = tokenizer(example_english_phrase, return_tensors="pt") >>> # Translate >>> generated_ids = model.generate(**inputs, num_beams=4, max_length=5) >>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] '42 este răspuns' ``` Mask filling example: ```python >>> from transformers import AutoTokenizer, MBartForConditionalGeneration >>> model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-cc25") >>> tokenizer = AutoTokenizer.from_pretrained("facebook/mbart-large-cc25") >>> # de_DE is the language symbol id for German >>> TXT = " Meine Freunde sind nett aber sie essen zu viel Kuchen. de_DE" >>> input_ids = tokenizer([TXT], add_special_tokens=False, return_tensors="pt")["input_ids"] >>> logits = model(input_ids).logits >>> masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item() >>> probs = logits[0, masked_index].softmax(dim=0) >>> values, predictions = probs.topk(5) >>> tokenizer.decode(predictions).split() ['nett', 'sehr', 'ganz', 'nicht', 'so'] ``` NzJThe `use_cache` argument is changed to `False` since `labels` is provided.F) rvrdrfrerr2rgrrhrr.r' losslogitsrrjrkrCrlrrm)rrhrwarningr;r*rrzrrrr!rrrjrkrCrlrrm)rHr)rvrdrerfrr2rgrrrhryoutputs lm_logitsmasked_lm_lossloss_fctoutputs r:rV%MBartForConditionalGeneration.forwardsr`&1%BF(,26:>*.!%#'z ##d*z  t+z !++d2 z !& 0 04 7 z uU%6%6784? z z ((4/z  %0047z   4'z $;z D[z +,z  5!2!23 3z z xDELLDDr<rz MBart model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cx^\rSrSrS\4U4Sjjr\\SS\RS-S\RS-S\RS-S\RS-S \ \RS-S \RS-S \RS-S \RS-S \ S-S\\S\\-4Sjj55rSrU=r$)MBartForSequenceClassificationirc >[TU]"U40UD6 [U5Ul[ UR UR UR UR5UlUR5 grj) rFrGrWrrr num_labelsclassifier_dropoutclassification_headr*)rHrryrIs r:rG'MBartForSequenceClassification.__init__sZ *6*' #: NN NN     % % $   r<Nr)rvrdrerfr2rgrrryrc UbSn Uc%Ub"[SURR35eUR"U4UUUUUUU S.U D6n U Sn UR UR R 5RU R5n [[R"U RS55R5S:HS5 XSS24RU RS5SU RS55SS2SSS24nUR!U5nSnUGbURUR5nUR R"cUR R$S:XaS UR lOyUR R$S:aNUR&[R(:XdUR&[R*:XaS UR lOS UR lUR R"S :XaS[-5nUR R$S:Xa&U"UR/5UR/55nOU"X5nOUR R"S :XaG[15nU"URSUR R$5URS55nO,UR R"S :Xa[35nU"X5n[5UUU R6U R8U R:U R<U R>U R@U RBS 9 $) aU decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Bart uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_bart._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NFz8Passing input embeddings is currently not supported for rvrdrerfr2rgrrr'z7All examples must have the same number of tokens.r. regressionsingle_label_classificationmulti_label_classificationr)"NotImplementedErrorrIrZreqr eos_token_idr5rOr#rQunique_consecutiver3numelrrr problem_typerrNrSr_rr6rrrrrjrkrCrlrrm)rHr)rvrdrerfr2rgrrryrreos_masksentence_representationrrrs r:rV&MBartForSequenceClassification.forward&sT  I  !:%J4>>KbKbJcd '+jj  ' )/#9+'"7 '  '   << 8 89<<]=Q=QR  $ $X\\!_ 5 ; ; = B E #0! "<"A"A-BTBTUVBWY[]j]o]opr]s"t r1H# ))*AB  YYv}}-F{{''/;;))Q./;DKK,[[++a/V\\UZZ5OSYS_S_chclclSl/LDKK,/KDKK,{{''<7"9;;))Q.#FNN$4fnn6FGD#F3D))-JJ+- B 0F0F GUWY))-II,./.#33")"?"?&99$55&-&G&G")"?"?&99  r<)rr) NNNNNNNNN)rZr[r\r]r(rGrrrQr?r`listr@rrrrrrVrarbrcs@r:rrs, { .2.259:>:>26:>*.!%i ##d*i  t+i !++d2 i !& 0 04 7 i e//047 i ((4/i  %0047i   4'i $;i +,i  0 0i i r<rc^\rSrSrU4Sjr\\SS\RS-S\RS-S\RS-S\RS-S\ \RS-S \RS-S \RS-S \RS-S \RS-S \ S-S\ \S\\-4Sjj55rSrU=r$)MBartForQuestionAnsweringic>[TU]U5 SUlURUl[U5Ul[ R "URUR5UlUR5 grC) rFrGrrWrrr hidden_size qa_outputsr*rs r:rG"MBartForQuestionAnswering.__init__s[   ++' ))F$6$68I8IJ r<Nr)rvrdrerfstart_positions end_positionsr2rgrryrc <UbUbSn UR"U4UUUUUU U S.U D6n U Sn URU 5nURSSS9unnURS5R 5nURS5R 5nSnUbUb[ UR 55S:aURS5n[ UR 55S:aURS5nUR S5nURSU5nURSU5n[US9nU"X5nU"UU5nUU-S - n[UUUU RU RU RU RU RU RU R S 9 $) aC decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Bart uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_bart._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. NFrrr'r.r,) ignore_indexrD) r start_logits end_logitsrrjrkrCrlrrm)rrsplitr6rrorrrrrrjrkrCrlrrm)rHr)rvrdrerfrrr2rgrryrsequence_outputrrr total_loss ignored_indexr start_lossend_losss r:rV!MBartForQuestionAnswering.forwardsP  &=+DI&*jj  ' )/#9+'"7 '  ' "!*1#)<<r<#: j#++B/::< ''+668   &=+D?'')*Q."1"9"9""==%%'(1, - 5 5b 9 (--a0M-33A}EO)//=AM']CH!,@J M:H$x/14J2%!#33")"?"?&99$55&-&G&G")"?"?&99  r<)rrrrr)rZr[r\r]rGrrrQr`r?rr@rrrrrrVrarbrcs@r:rrs< *..259:>:>371526:>!%W <<$&W  t+W !++d2 W !& 0 04 7 W e//047 W ))D0W ''$.W ((4/W  %0047W $;W +,W  4 4W W r<rc2^\rSrSrSrU4SjrSrSrU=r$)MBartDecoderWrapperiz This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is used in combination with the [`EncoderDecoderModel`] framework. cd>[TU]U5 [U5UlUR 5 grj)rFrGrBr[r*rs r:rGMBartDecoderWrapper.__init__s&  #F+  r<c&UR"U0UD6$rjr[)rHargsrys r:rVMBartDecoderWrapper.forward s||T,V,,r<r) rZr[r\r]r^rGrVrarbrcs@r:rrs  --r<rcj^\rSrSrSS0rU4SjrSrSr\\ SS\ RS-S \ RS-S \ RS-S \ RS-S \S-S \ RS-S\ RS-S\S-S\\ R-S\\S\\-4Sjj55rSrU=r$)MBartForCausalLMirvz!model.decoder.embed_tokens.weightc>SUlSUl[TU] U5 [ U5Ul[ R"URURSS9Ul UR5 g)NTFr) rrKrFrGrrrrrr!rzr*rs r:rGMBartForCausalLM.__init__sX $)!  (0 yy!3!3V5F5FUS  r<cBURRR$rjrr[r"r/s r:r^%MBartForCausalLM.get_input_embeddings!szz!!...r<c8XRRlgrjrras r:rb%MBartForCausalLM.set_input_embeddings$s*/ 'r<Nr)rvrrrr2rrlogits_to_keepryrc URR"SUUUUUUUS.U D6n U Sn [U [5(a [ U *S5OU n UR U SS2U SS245nSnUbaUR UR5n[5nU"URSURR5URS55n[UUU RU RU RU R S9$)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, MBartForCausalLM >>> tokenizer = AutoTokenizer.from_pretrained("facebook/mbart-large-cc25") >>> model = MBartForCausalLM.from_pretrained("facebook/mbart-large-cc25") >>> assert model.config.is_decoder, f"{model.__class__} has to be configured as a decoder." >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> expected_shape = [1, inputs.input_ids.shape[-1], model.config.vocab_size] >>> list(logits.shape) == expected_shape True ```)r)rvrrrr2rrNr.)rrrrrrCr)rr[rr_slicerzr5rOrrrr!rrrrrC)rHr)rvrrrr2rrrryrr slice_indicesrrrs r:rVMBartForCausalLM.forward'sL>BZZ=O=O > )"7#9+' >  >   8B>SV8W8W~ot4]k mA}a,?@A  YYv}}-F')HFKKDKK,B,BCV[[QS_UD0#33!//))$55   r<r) NNNNNNNNr)rZr[r\r]rsrGr^rbrrrQr?r`r@r rr_rrrrrVrarbrcs@r:rrs/= /0.2.2:>;?(,26*.!%-.A ##d*A  t+A  %0047 A !& 1 1D 8 A  A ((4/A   4'A $;A ell*A +,A  2 2A A r<r)rrrrrWr)Nr)Rr^rcollections.abcrrQrtorch.nnrrrr r activationsr cache_utilsr r r generationr masking_utilsrrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrrrrmodeling_utilsrrprocessing_utilsrutilsrrrr r!r"r# utils.genericr$utils.output_capturingr%r&configuration_mbartr( get_loggerrZrr`r_r;rr>reModulerqrrrrrrrrBrWrrrrr__all__rr<r:rsz $ AA&!CC)J:G&8E, !!   H %%,,c*;bll;8 =r|| =(!% II% <<% % << % LL4' % T\ %%'(%:r)RYYr)j525pZ2Z|bii0?4r@'r@jh 'h Vp %p p f \D$8/\D  \D~ z %9z  z zg 4g g V -. - Y +_Y x r<