ZjPSrSSKrSSKrSSKJr SSKrSSKJr SSKJrJ r J r SSK J r SSKJr SS KJrJrJr SS KJr SS KJrJr SS KJr SS KJr SSKJrJrJ r J!r!J"r"J#r#J$r$ SSK%J&r&J'r' SSK(J)r) SSK*J+r+J,r,J-r-J.r.J/r/J0r0 SSK1J2r2 SSK3J4r4J5r5 SSK6J7r7 \/Rp"\95r:S\RvS\\H\55rN\,"S?S<9"S@SA\H55rO\,"SBSC\H55rP"SDSE\H5rQ\,"SFS<9"SGSH\H\55rR/SIQrSg)KzPyTorch BART model.N)Callable)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleis_torchdynamo_compilingloggingtorch_compilable_check)merge_with_config_defaults)OutputRecordercapture_outputs) BartConfig input_ids pad_token_iddecoder_start_token_idcURUR5nUSS2SS24R5USS2SS24'X#SS2S4'Uc [S5eUR US:HU5 U$)z) Shift input ids one token to the right. Nr&rz1self.model.config.pad_token_id has to be defined.i) new_zerosshapeclone ValueError masked_fill_)r(r)r*shifted_input_idss w/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/bart/modeling_bart.pyshift_tokens_rightr4:sz"++IOO<(CRC0668ae4adLMM""#4#SUl[TU] XR-U5 gN)offsetsuper__init__)selfr9r: __class__s r3r@'BartLearnedPositionalEmbedding.__init__Os"  ++5}Er5Nr(past_key_values_length position_idsc>Uc]URSSupE[R"X"U-[RURR S9R US5nOURS5n[TU]%X0R-5$)z3`input_ids' shape is expected to be [bsz x seqlen].Nr=)dtypedevicer,r) r.torcharangelongweightrHexpand unsqueezer?forwardr>)rAr(rDrEbszseq_lenrBs r3rO&BartLearnedPositionalEmbedding.forwardUs  $??2A.LC <<&(HPUPZPZcgcncncucufS"o (11!4Lw|kk9::r5)r>)rN) __name__ __module__ __qualname____firstlineno____doc__intr@rITensorrO__static_attributes__ __classcell__rBs@r3r7r7JsWFsF3Fmq ; ;?B ;V[VbVbeiVi ; ;r5r7c r^\rSrSrSrS S\S\S\S\S-4U4SjjjrS \R4U4S jjr S r U=r $) BartScaledWordEmbeddingezT This module overrides nn.Embeddings' forward by multiplying with embeddings scale. r9r: padding_idx embed_scaleNc2>[TU]XU5 X@lgN)r?r@ra)rAr9r:r`rarBs r3r@ BartScaledWordEmbedding.__init__js  D&r5r(c<>[TU]U5UR-$rc)r?rOra)rAr(rBs r3rOBartScaledWordEmbedding.forwardnswy)D,<,<<L==((6??([L,,|3K''1-88:K  $$r5c$^\rSrSrSrSS\S\S\S\S\S \S \S-S \S-4U4S jjjr SS \ RS\ RS-S\ S-S\ RS-S\ \S\\ R\ RS-44 SjjrSrU=r$) BartAttentionz=Multi-headed attention from 'Attention Is All You Need' paperN embed_dim num_headsrq is_decoderbias is_causalconfig layer_idxc t>[T U]5 XlX lX0lX-UlXplUR U-UR:wa[SURSUS35eUR S-UlX@l X`l Xl Uc>UR(a-[RSURRS35 [ R""XUS9Ul[ R""XUS9Ul[ R""XUS9Ul[ R""XUS9Ulg)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).rtzInstantiating a decoder z without passing `layer_idx` is not recommended and will lead to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.r)r?r@rrrqhead_dimrr0rprrrlogger warning_oncerBrSrLineark_projv_projq_projout_proj) rArrrqrrrrrrBs r3r@BartAttention.__init__s  "" !.  MMI %$.. 8MdnnM]$YKr3 }}d* $""     *4>>+B+B*CD,,  ii 4@ ii 4@ ii 4@  )TB r5 hidden_stateskey_value_statespast_key_valuesrorrreturnc ,USLnURSSn/UQSPURP7nURU5RU5R SS5n Sn Ub][ U[ 5(aFURRUR5n U(a URn OURn OUn U(aUOUn U(aQUbNU (aGW RURRn U RURRnOURU 5n UR!U 5n/U RSSQSPURP7nU RU5R SS5n URU5R SS5nUbSW R#XUR5upU(a.[ U[ 5(aSURUR'[$R&"UR(R*[,5nU"UU U UU4UR.(dSO UR0UR2S.UD6unnUR4"/UQSP76R75nUR9U5nUU4$) z#Input shape: Batch x Time x ChannelNr,r&r=FT)rqrp)r.rrviewr| isinstancer is_updatedgetrcross_attention_cacheself_attention_cachelayerskeysvaluesrrupdater get_interfacer_attn_implementationrryrqrpreshaperr)rArrrrorris_cross_attention input_shape hidden_shape query_statesrcurr_past_key_valuescurrent_states key_states value_stateskv_shapeattention_interfacerrs r3rOBartAttention.forwardsc.T9$))#2. 88b8$--8 {{=166|DNNqRST   &/+>??,77;;DNNK %+:+P+P(+:+O+O('6$-?)] /"=*-44T^^DIIJ/66t~~FMML^4J;;~6LF--cr2FBF FH#2<(?(M(M KK , ,.E) %8      % $}}C$,,LL %  % ! \"));;;;FFH mmK0 L((r5) rrqrrrrrrrrrrpr)rFTFNNNNN)rSrTrUrVrWrXrjboolr'r@rIrYr rrtuplerOrZr[r\s@r3rrsG  $( $%C%C%C %C  %C  %C%CT!%C:%C%CT15(,.2 H)||H) ,,-H) H)  t+ H)-.H) u||U\\D00 1H)H)r5rc^\rSrSrS S\S\S-4U4SjjjrS\RS\RS\ \ S \R4S jr S r U=r$) BartEncoderLayeriNrrcl>[TU]5 URUl[ URUR UR UUS9Ul[R"UR5Ul URUl [URUlURUl[R "URUR"5Ul[R "UR"UR5Ul[R"UR5Ulg)N)rrrqrr)r?r@d_modelrrencoder_attention_headsattention_dropout self_attnr LayerNormself_attn_layer_normrqr activation_function activation_fnactivation_dropoutrencoder_ffn_dimfc1fc2final_layer_normrArrrBs r3r@BartEncoderLayer.__init__s &nn44,,  %'LL$@!~~ #F$>$>?"(";";99T^^V-C-CD99V33T^^D " T^^ <r5rrorrrc 2UnUR"U4SU0UD6up[RRXRURS9nXA-nUR U5nUnUR URU55n[RRXRURS9nURU5n[RRXRURS9nXA-nURU5nUR[R:Xal[R"U5R5(dC[R "UR5R"S- n[R$"X*US9nU$)Nrorwi)minmax)rrr}rqryrrrrrrrGrIfloat16isfiniteallfinforclamp)rArrorrresidual_ clamp_values r3rOBartEncoderLayer.forwards[ !>>  )    --m||VZVcVc-d  0 11-@  **488M+BC  --m?V?Vaeanan-o /  --m||VZVcVc-d  0 --m<   %-- /}8U8Y8Y8[8[++m&9&9:>>EK!KK <[YMr5) rrrqrrrrrrrc)rSrTrUrVr'rXr@rI FloatTensorrrrYrOrZr[r\s@r3rrse=z=cDj==&(())+,   r5rc^\rSrSrSS\S\S-4U4SjjjrSS\RS\RS-S\RS-S \RS-S \ S-S \ S-S \ \ S \R4Sjjr SrU=r$)BartDecoderLayeri7Nrrc >[TU]5 URUl[ URUR UR SSUUS9UlURUl[URUl URUl [R"UR5Ul[ URUR UR SUUS9Ul[R"UR5Ul[R$"URUR&5Ul[R$"UR&UR5Ul[R"UR5Ulg)NT)rrrqrrrr)rqrrr)r?r@rrrdecoder_attention_headsrrrqr rrrrrr encoder_attnencoder_attn_layer_normrdecoder_ffn_dimrrrrs r3r@BartDecoderLayer.__init__8s  &nn44,, ~~ #F$>$>?"(";";$&LL$@!) NN  * *,,  (*||DNN'C$99T^^V-C-CD99V33T^^D " T^^ <r5rroencoder_hidden_statesencoder_attention_maskr use_cacherrrc UnUR"U4UUS.UD6up[RRXRURS9nX-nUR U5nUbbUnUR "U4UUUS.UD6up[RRXRURS9nX-nURU5nUnURURU55n[RRXRURS9nURU5n[RRXRURS9nX-nURU5nU$)N)rrorw)rror) rrr}rqryrrrrrrrr) rArrorrrrrrrrs r3rOBartDecoderLayer.forwardWs! >>  +)     --m||VZVcVc-d  0 11-@  ! ,$H#00 !65 /      MMM11-<[TU]5 [R"X5Ul[R "US9Ul[R"X#5Ulg)N)rx)r?r@rrdenseDropoutrqr)rArrrrrBs r3r@BartClassificationHead.__init__s@ YYy4 zzN3  )9 r5rrcURU5nURU5n[R"U5nURU5nUR U5nU$rc)rqrrItanhr)rArs r3rOBartClassificationHead.forwardsN ]3  =1  =1  ]3  m4 r5)rrqrrir\s@r3rrsQ7 : : : :  :U\\ellr5rcp^\rSrSr%\\S'SrSrSS/rSS/r S r Sr Sr Sr SrU4S jr\S 5rS rU=r$) BartPreTrainedModelirmodelTzencoder.versionzdecoder.versionrrrc>[TU]U5 [U[5(a![R "UR 5 ggrc)r? _init_weightsrBartForConditionalGenerationinitzeros_final_logits_bias)rArkrBs r3r!BartPreTrainedModel._init_weightss5 f% f: ; ; KK00 1 [TU]U5 URUlURUlUR nUR UlURUl UR(a[R"U5OSn[URX RUS9Ul[!URU5Ul[$R&"[)UR*5Vs/sH n[-XS9PM sn5Ul[$R0"U5UlSUlUR75 gs snfNrhrg)rF)r?r@rqencoder_layerdrop layerdroprr)r`max_position_embeddingsmax_source_positionsscale_embeddingmathsqrtr^ vocab_size embed_tokensr7embed_positionsr ModuleListrangeencoder_layersrrrlayernorm_embeddinggradient_checkpointing post_init)rArrrairBs r3r@BartEncoder.__init__s  ~~ 11NN !..$*$B$B!.4.D.Ddii *# 3   y*:*:   >  * *  mmTYZ`ZoZoTp$qTpq%5f%JTp$qr #%<< #: &+#  %rs0ENr(ro inputs_embedsrrrc hUSLUSL- (a [S5eUcURU5nURUSS2SS2S45nURUR5nX5-nUR U5n[ RRX`RURS9n[URUUS9n[UR5HRupxSn UR(a'[R"/5n XR :aSn U (aMHU"UU40UD6nMT [#US9$)Nz:You must specify exactly one of input_ids or inputs_embedsr,rw)rr=roFT)last_hidden_state)r0r3r4torHr8rr}rqryrr enumeraterrIrandr,r) rAr(ror=rr embed_posridx encoder_layerto_dropdropout_probabilitys r3rOBartEncoder.forwards,  -t"; <YZ Z   --i8M((q!Rx)@A LL!5!56 %1 00?  --m||VZVcVc-d 2;;')  #,DKK"8 CG}}&+jjn#&7"G7 -!"!! #9+  r5) rqr4r3r9r,r8rr.r`r)rSrTrUrVrWrr_can_record_outputsr'r@r#r%rrI LongTensorrYrrrrrOrZr[r\s@r3r'r's*# z4 .2.226 * ##d**  t+* ((4/ * +, *  *  * r5r'cJ^\rSrSrSr\\"\SSS9\"\SSS9S.rS\ 4U4S jjr \ \ \ SS \RS -S \R S -S \R"S -S\RS -S\S -S\R"S -S\S -S\\S\4Sjj555rSrU=r$) BartDecoderi(z Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`BartDecoderLayer`] Args: config: BartConfig embed_tokens (nn.Embedding): output embedding r&r)index layer_namer)rr(cross_attentionsrc >[TU]U5 URUlURUlUR UlURUlUR(a [R"UR5OSn[URURUR US9Ul[!URUR5Ul[$R&"[)UR*5Vs/sH n[-XS9PM sn5Ul[$R0"UR5UlSUlUR75 gs snfr*)r?r@rqdecoder_layerdropr,r)r`r-max_target_positionsr/r0r1rr^r2r3r7r4rr5r6decoder_layersrrrr8r9r:)rArrar;rBs r3r@BartDecoder.__init__7s  ~~ 11!..$*$B$B!393I3Idii/s 3   v~~t/?/?[  >  * * NN mmTYZ`ZoZoTp$qTpq%5f%JTp$qr #%<<#? &+#  %rsENr(rorrrr=rrrrc USLUSL- (a [S5eUcURU5nU(abUc_UcURR(a.[ [ URS9[ URS95O[ URS9nUR 5SSupUbUR5OSn [R"XRS9U -n Uc2[5(d#X-n [R"XURS9n[U[5(a UROUn[URUUUS9n[!URUUUS9nUR#[$XS9nUR'UR5nXo-nUR)U5n[*R,R/UUR.UR0S 9n[3UR45HNunnUR0(a([R6"/5nUUR8:aM?U"UUU4UUUS .UD6nMP [;UUS 9$) NzJYou must specify exactly one of decoder_input_ids or decoder_inputs_embeds)rr,rr)rr=ror)rr=ror)rErw)rrr)r?r)r0r3ris_encoder_decoderr r rzget_seq_lengthrIrJrHr onesrrrrr4inputr@r8rr}rqryrArrBr,r)rAr(rorrrr=rrr batch_size seq_lengthrDrEmask_seq_lengthself_attn_cache positionsrrD decoder_layerrGs r3rOBartDecoder.forwardOsD  -t"; <ij j   --i8M 0)4 8V8V$L $DlZ^ZeZeFfg!5  "/!3!3!5cr!: ETE`!?!?!Afg||J7K7KLOee  !*B*D*D4AO"ZZ ML`L`aN/+>??  0 0   ,;;')+   ";;;'1"7 " ((0F(b LL!5!56 %1 00?  --mt||VZVcVc-d "+DKK"8 C}}&+jjn#&7)%(> /# M#9"9++  r5) rqr4r3r9r,r8rrRr`)NNNNNNN)rSrTrUrVrWrr$rrIr'r@r#r%rrIrJrYrr rrrrrOrZr[r\s@r3rLrL(s*$]! T*=n] z0 .2.2:>:>(,26!%R ##d*R  t+R  %0047 R !& 0 04 7 R  R ((4/R $;R +,R  3R  R r5rLcz^\rSrSrSSS.rS\4U4SjjrSrSr\ \ SS \ RS-S \ RS-S \ RS-S \ RS-S \\ RS-S\S-S\ RS-S\ RS-S\S-S\\S\\-4Sjj55rSrU=r$) BartModeliz shared.weight)zdecoder.embed_tokens.weightzencoder.embed_tokens.weightrcJ>[TU]U5 URURp2UR(a [ R "UR5OSn[X1RX$S9Ul [U5Ul [U5Ul UR5 g)Nrhrg)r?r@r)r2r/r0r1rr^sharedr'encoderrLdecoderr:)rArr`r2rarBs r3r@BartModel.__init__su  "("5"5v7H7HZ393I3Idii/s -j..+o "6* "6*  r5cUR$rc)rdrs r3get_input_embeddingsBartModel.get_input_embeddingss {{r5c|XlURURlURURlgrc)rdrer3rfrArns r3set_input_embeddingsBartModel.set_input_embeddingss' $(KK !$(KK !r5Nr(rodecoder_input_idsdecoder_attention_maskencoder_outputsrr=decoder_inputs_embedsrrrrc VUcEUcBUc [S5e[XRRURR5nUcUR "S UUUS.U D6nOK[ U[5(d6[US[U5S:aUSOS[U5S:aUSOSS9nUR"S UUUSUUUU S.U D6n [U RU RU RU RU RURURURS 9$) C decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Bart uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_bart._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. NzIf no `decoder_input_ids` or `decoder_inputs_embeds` are passed, `input_ids` cannot be `None`. Please pass either `input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.)r(ror=rr&r=)r?rr(r(rorrrr=r)r?rdecoder_hidden_statesdecoder_attentionsrOencoder_last_hidden_staterencoder_attentionsr)r0r4rr)r*rerrlenrfrr?rrr(rO) rAr(rororprqrr=rrrrrdecoder_outputss r3rOBartModel.forwardsaP  $)>)F  U !3;;33T[[5W5W!   "/3||0#-+0 0O O_==-"1!"4474H14Loa0RV14_1E1I?1-tO FJ\\ F '1"1!"4#1+/ F  F "-??+;;"1"?"?.99,==&5&G&G"1"?"?.99  r5)rfrerd NNNNNNNNN)rSrTrUrV_tied_weights_keysr'r@rirmrrrIrJrYlistrr rrrrrrOrZr[r\s@r3rbrbs?(7'6 z 0 .2.259:>:>(,26:>!%T ##d*T  t+T !++d2 T !& 0 04 7 T e//047 T T ((4/T  %0047T $;T +,T  # #T T r5rbzV The BART Model with a language modeling head. Can be used for summarization. ) custom_introc ^\rSrSrSrSS0rS/rS\4U4SjjrSS \ S \ S-S \ S \ R4U4S jjjr S \ S S4Sjr\\SS\R$S-S\R&S-S\R$S-S\R$S-S\\R*S-S\S-S\R*S-S\R*S-S\R$S-S\ S-S\\S \\-4Sjj55rS\R&4SjrSrU=r$)rirlm_head.weightzmodel.shared.weightrrcv>[TU]U5 [U5UlUR S[ R "SURRR455 [R"URURRRSS9Ul UR5 g)Nrr&Fr)r?r@rbrregister_bufferrIzerosrdr9rrrlm_headr:rArrBs r3r@%BartForConditionalGeneration.__init__(s  v&  0%++q$**BSBSBbBb>c2deyy1B1B1Q1QX]^  r5Nnew_num_tokenspad_to_multiple_of mean_resizingrcx>[TU]XU5nURURRS5 U$)Nr)r?resize_token_embeddings_resize_final_logits_biasrLr.)rArrrnew_embeddingsrBs r3r4BartForConditionalGeneration.resize_token_embeddings1s<8]jk &&~'<'<'B'B1'EFr5c,URRSnX::aURSS2SU24nON[R"SX- 4URRS9n[R "URU/SS9nUR SU5 g)Nr,r&rrur)rr.rIrrHcatr)rArold_num_tokensnew_bias extra_biass r3r6BartForConditionalGeneration._resize_final_logits_bias8s//55b9  +--a..@AHa)H%IRVRhRhRoRopJyy$"8"8*!E1MH 0(;r5r(rororprqrr=rrlabelsrrrc U bXU (a[RS5 Sn Uc7Uc4[XRRURR 5nUR "U4UUUUUUUU S.U D6n URU S5n XRRU R5-n SnU baU RU R5n [5nU"U RSURR5U RS55n[UU U RU R U R"U R$U R&U R(U R*S9 $)ap decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Bart uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_bart._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example summarization: ```python >>> from transformers import AutoTokenizer, BartForConditionalGeneration >>> model = BartForConditionalGeneration.from_pretrained("facebook/bart-large-cnn") >>> tokenizer = AutoTokenizer.from_pretrained("facebook/bart-large-cnn") >>> ARTICLE_TO_SUMMARIZE = ( ... "PG&E stated it scheduled the blackouts in response to forecasts for high winds " ... "amid dry conditions. The aim is to reduce the risk of wildfires. Nearly 800 thousand customers were " ... "scheduled to be affected by the shutoffs which were expected to last through at least midday tomorrow." ... ) >>> inputs = tokenizer([ARTICLE_TO_SUMMARIZE], max_length=1024, return_tensors="pt") >>> # Generate Summary >>> summary_ids = model.generate(inputs["input_ids"], num_beams=2, min_length=0, max_length=20) >>> tokenizer.batch_decode(summary_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] 'PG&E scheduled the blackouts in response to forecasts for high winds amid dry conditions' ``` Mask filling example: ```python >>> from transformers import AutoTokenizer, BartForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("facebook/bart-base") >>> model = BartForConditionalGeneration.from_pretrained("facebook/bart-base") >>> TXT = "My friends are but they eat too many carbs." >>> input_ids = tokenizer([TXT], return_tensors="pt")["input_ids"] >>> logits = model(input_ids).logits >>> masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item() >>> probs = logits[0, masked_index].softmax(dim=0) >>> values, predictions = probs.topk(5) >>> tokenizer.decode(predictions).split() ['not', 'good', 'healthy', 'great', 'very'] ``` NzJThe `use_cache` argument is changed to `False` since `labels` is provided.F)rororqrprr=rrrrr, losslogitsrrvrwrOrxrry)rwarningr4rr)r*rrrr@rHrrr2rrrvrwrOrxrry)rAr(rororprqrr=rrrrrroutputs lm_logitsmasked_lm_lossloss_fcts r3rO$BartForConditionalGeneration.forwardAsdh  klI (-B-J$6KK44dkk6X6X%!'+jj  ' )/+#9+'"7 '  ' LL,  6 6 9 9):J:J KK   YYy//0F')H%innR9O9O&PRXR]R]^`RabN#33")"?"?&99$55&-&G&G")"?"?&99  r5cj[XRRURR5$rc)r4rr)r*)rArs r3%prepare_decoder_input_ids_from_labelsBBartForConditionalGeneration.prepare_decoder_input_ids_from_labelss#!&++*B*BDKKDfDfggr5rr)NT NNNNNNNNNN)rSrTrUrVr r~_keys_to_ignore_on_load_missingr'r@rXrr EmbeddingrrrrrIrJrYrrr rrrrrOrrZr[r\s@r3rrs  /(;&;#zae!7:TzY] <<<.2.259:>:>(,26:>*.!%{ ##d*{  t+{ !++d2 { !& 0 04 7 { e//047 { { ((4/{  %0047{   4'{ $;{ +,{   { { zhELLhhr5rz Bart model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cx^\rSrSrS\4U4Sjjr\\SS\RS-S\RS-S\RS-S\RS-S \ \RS-S \RS-S \RS-S \RS-S \ S-S\\S\\-4Sjj55rSrU=r$)BartForSequenceClassificationirc >[TU]"U40UD6 [U5Ul[ UR UR UR UR5UlUR5 grc) r?r@rbrrr num_labelsclassifier_dropoutclassification_headr:)rArrrrBs r3r@&BartForSequenceClassification.__init__sZ *6*v& #9 NN NN     % % $   r5Nr(rororprqr=rrrrrrrc UbSn Uc%Ub"[SURR35eUR"U4UUUUUUU S.U D6n U Sn UR UR R 5RU R5n [[R"U RS55R5S:HS5 XSS24RU RS5SU RS55SS2SSS24nUR!U5nSnUGbURUR5nUR R"cUR R$S:XaS UR lOyUR R$S:aNUR&[R(:XdUR&[R*:XaS UR lOS UR lUR R"S :XaS[-5nUR R$S:Xa&U"UR/5UR/55nOU"X5nOUR R"S :XaG[15nU"URSUR R$5URS55nO,UR R"S :Xa[35nU"X5n[5UUU R6U R8U R:U R<U R>U R@U RBS 9 $) aU decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Bart uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_bart._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NFz8Passing input embeddings is currently not supported for rororprqr=rrrrr&z7All examples must have the same number of tokens.r, regressionsingle_label_classificationmulti_label_classificationr)"NotImplementedErrorrBrSreqr eos_token_idr@rHr"rIunique_consecutivesumnumelrrzr problem_typerrGrKrXrsqueezerrrrrvrwrOrxrry)rAr(rororprqr=rrrrrrrreos_masksentence_representationrrrs r3rO%BartForSequenceClassification.forwardsR  I  !:%J4>>KbKbJcd '+jj  ' )/#9+'"7 '  '   << 8 89<<]=Q=QR  $ $X\\!_ 5 ; ; = B E #0! "<"A"A-BTBTUVBWY[]j]o]opr]s"t r1H# ))*AB  YYv}}-F{{''/;;))Q./;DKK,[[++a/V\\UZZ5OSYS_S_chclclSl/LDKK,/KDKK,{{''<7"9;;))Q.#FNN$4fnn6FGD#F3D))-JJ+- B 0F0F GUWY))-II,./.#33")"?"?&99$55&-&G&G")"?"?&99  r5)rrr})rSrTrUrVr'r@rrrIrJrYrrrrrrrrOrZr[r\s@r3rrs, z .2.259:>:>26:>*.!%i ##d*i  t+i !++d2 i !& 0 04 7 i e//047 i ((4/i  %0047i   4'i $;i +,i  0 0i i r5rc^\rSrSrU4Sjr\\SS\RS-S\RS-S\RS-S\RS-S\ \RS-S \RS-S \RS-S \RS-S \RS-S \ S-S\ \S\\-4Sjj55rSrU=r$)BartForQuestionAnsweringiFc>[TU]U5 SUlURUl[U5Ul[ R "URUR5UlUR5 gr<) r?r@rrbrrr hidden_size qa_outputsr:rs r3r@!BartForQuestionAnswering.__init__Hs[   ++v& ))F$6$68I8IJ r5Nr(rororprqstart_positions end_positionsr=rrrrrrc <UbUbSn UR"U4UUUUUU U S.U D6n U Sn URU 5nURSSS9unnURS5R 5nURS5R 5nSnUbUb[ UR 55S:aURS5n[ UR 55S:aURS5nUR S5nURSU5nURSU5n[US9nU"X5nU"UU5nUU-S - n[UUUU RU RU RU RU RU RU R S 9 $) rtNFrrr&r,ru) ignore_indexr=) r start_logits end_logitsrrvrwrOrxrry)rrsplitrrrzrzrrrrrvrwrOrxrry)rAr(rororprqrrr=rrrrrrsequence_outputrrr total_loss ignored_indexr start_lossend_losss r3rO BartForQuestionAnswering.forwardTsN  &=+DI&*jj  ' )/#9+'"7 '  ' "!*1#)<<r<#: j#++B/::< ''+668   &=+D?'')*Q."1"9"9""==%%'(1, - 5 5b 9 (--a0M-33A}EO)//=AM']CH!,@J M:H$x/14J2%!#33")"?"?&99$55&-&G&G")"?"?&99  r5)rrrr)rSrTrUrVr@rrrIrYrJrrrrrrrrOrZr[r\s@r3rrFs< *..259:>:>371526:>!%W <<$&W  t+W !++d2 W !& 0 04 7 W e//047 W ))D0W ''$.W ((4/W  %0047W $;W +,W  4 4W W r5rc2^\rSrSrSrU4SjrSrSrU=r$)BartDecoderWrapperiz This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is used in combination with the [`EncoderDecoderModel`] framework. cd>[TU]U5 [U5UlUR 5 grc)r?r@rLrfr:rs r3r@BartDecoderWrapper.__init__s&  "6*  r5c&UR"U0UD6$rcrf)rAargsrrs r3rOBartDecoderWrapper.forwards||T,V,,r5r) rSrTrUrVrWr@rOrZr[r\s@r3rrs  --r5rzu BART decoder with a language modeling head on top (linear layer with weights tied to the input embeddings). cj^\rSrSrSS0rU4SjrSrSr\\ SS\ RS-S \ RS-S \ RS-S \ RS-S \S-S \ RS-S\ RS-S\S-S\\ R-S\\S\\-4Sjj55rSrU=r$)BartForCausalLMirz!model.decoder.embed_tokens.weightc>SUlSUl[TU] U5 [ U5Ul[ R"URURSS9Ul UR5 g)NTFr) rrVr?r@rrrrrr2rr:rs r3r@BartForCausalLM.__init__sX $)!  '/ yy!3!3V5F5FUS  r5cBURRR$rcrrfr3rs r3ri$BartForCausalLM.get_input_embeddingsszz!!...r5c8XRRlgrcrrls r3rm$BartForCausalLM.set_input_embeddingss*/ 'r5Nr(rorrrr=rrlogits_to_keeprrrc URR"SUUUUUUUS.U D6n U Sn [U [5(a [ U *S5OU n UR U SS2U SS245nSnUbaUR UR5n[5nU"URSURR5URS55n[UUU RU RU RU R S9$)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, BartForCausalLM >>> tokenizer = AutoTokenizer.from_pretrained("facebook/bart-base") >>> model = BartForCausalLM.from_pretrained("facebook/bart-base") >>> assert model.config.is_decoder, f"{model.__class__} has to be configured as a decoder." >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> expected_shape = [1, inputs.input_ids.shape[-1], model.config.vocab_size] >>> list(logits.shape) == expected_shape True ```rurNr,)rrrrr(rOr)rrfrrXslicerr@rHrrrr2rrrr(rO)rAr(rorrrr=rrrrrrr slice_indicesrrrs r3rOBartForCausalLM.forwardsL>BZZ=O=O > )"7#9+' >  >   8B>SV8W8W~ot4]k mA}a,?@A  YYv}}-F')HFKKDKK,B,BCV[[QS_UD0#33!//))$55   r5r) NNNNNNNNr)rSrTrUrVr~r@rirmrrrIrJrYrr rrXrrrrrOrZr[r\s@r3rrs1 = /0.2.2:>;?(,26*.!%-.A ##d*A  t+A  %0047 A !& 1 1D 8 A  A ((4/A   4'A $;A ell*A +,A  2 2A A r5r)rrrrrbrr#r)Nr)TrWr0rcollections.abcrrIrtorch.nnrrrr r activationsr cache_utilsr r r generationr masking_utilsrrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrrrrmodeling_utilsrrprocessing_utilsrutilsrrrr r!r" utils.genericr#utils.output_capturingr$r%configuration_bartr' get_loggerrSrrYrXr4rr7r^Modulerjrrrrrrrr#r'rLrbrrrrr__all__rr5r3rs $ AA&!CC)JB9G&8E*   H % %,, c [^  ;R\\;6 =bll =(!% II% <<% % << % LL4' % T\ %%'(%8r)BIIr)j010fO1OdRYY0/: -  - V %V r| %| ~q #q q h `h#6`h  `hF y $7y  y xf 2f f R -, - Y )?Y  Y x r5