ZjCSrSSKrSSKrSSKJr SSKJr SSKJr SSK J r SSK J r SS K JrJrJr SS KJr SS KJr SS KJrJrJr SS KJr SSKJr SSKJrJrJ r J!r!J"r" SSK#J$r$ SSK%J&r& SSK'J(r( "SS\"5r)\"SS\55r*"SS\ 5r+"SS\5r,\"SS\*55r-\"SS 9"S!S"\*\ 55r."S#S$\5r/"S%S&\$5r0"S'S(\!5r1/S)Qr2g)*zPyTorch PLBART model.N)nn)CrossEntropyLoss)initialization)Cache)GenerationMixin)BaseModelOutputSeq2SeqLMOutputSeq2SeqModelOutput)PreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuple)merge_with_config_defaults)capture_outputs)BartClassificationHead BartDecoder BartEncoderBartForCausalLMBartScaledWordEmbedding)'BigBirdPegasusForSequenceClassification)shift_tokens_right) PLBartConfigc\rSrSrSrg)PLBartScaledWordEmbedding/N__name__ __module__ __qualname____firstlineno____static_attributes__r z/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/plbart/modular_plbart.pyrr/r'rcP^\rSrSr%\\S'SrSrSS/rSr Sr Sr U4Sjr Sr U=r$) PLBartPreTrainedModel3configmodelTPLBartDecoderLayerPLBartEncoderLayerc>[TU]U5 [U[5(a![R "UR 5 ggN)super _init_weights isinstancePLBartForConditionalGenerationinitzeros_final_logits_bias)selfmodule __class__s r(r4#PLBartPreTrainedModel._init_weights=s5 f% f< = = KK00 1 >r'r )r"r#r$r%r__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_flash_attn_supports_sdpa_supports_flex_attnr4r& __classcell__r<s@r(r+r+3s< &*#-/CDN22r'r+c\rSrSrSrg) PLBartEncoderCr Nr!r r'r(rHrHCr)r'rHc\rSrSrSrg) PLBartDecoderGr Nr!r r'r(rKrKGr)r'rKc^\rSrSrSSS.rS\4U4SjjrSrSr\ \ \ SS \ RS-S \ RS-S \ RS-S \ RS-S \\ R S-S\S-S\ R S-S\ R S-S\S-S\\S\\ R\-4Sjj555rSrU=r$) PLBartModelKz shared.weight)zencoder.embed_tokens.weightzdecoder.embed_tokens.weightr-cJ>[TU]U5 URURp2UR(a [ R "UR5OSn[X1RX$S9Ul [U5Ul [U5Ul UR5 g)Ng?) embed_scale)r3__init__ pad_token_id vocab_sizescale_embeddingmathsqrtd_modelrsharedrHencoderrKdecoder post_init)r:r- padding_idxrTrQr<s r(rRPLBartModel.__init__Rss  "("5"5v7H7HZ393I3Idii/s / NNKq $V, $V,  r'cUR$r2)rY)r:s r(get_input_embeddings PLBartModel.get_input_embeddings^s {{r'c|XlURURlURURlgr2)rYrZ embed_tokensr[)r:values r(set_input_embeddings PLBartModel.set_input_embeddingsas' $(KK !$(KK !r'N input_idsattention_maskdecoder_input_idsdecoder_attention_maskencoder_outputspast_key_values inputs_embedsdecoder_inputs_embeds use_cachekwargsreturnc Uc"Uc[XRR5nUcUR"S UUUS.U D6nOK[ U[ 5(d6[ US[ U5S:aUSOS[ U5S:aUSOSS9nUR"S UUUSUUUU S.U D6n [U RU RU RU RU RURURURS9$) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`] or [`PLBartMultiTokenizer`] depending on the checkpoint. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) PLBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 50003 for *en_XX*, and 50001 for *java*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (: obj:*torch.LongTensor* of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. N)rgrhrmrrr)last_hidden_state hidden_states attentions)rgrhencoder_hidden_statesencoder_attention_maskrlrmro)rsrldecoder_hidden_statesdecoder_attentionscross_attentionsencoder_last_hidden_statervencoder_attentionsr )rr-rSrZr5r lenr[r rsrlrtrurz) r:rgrhrirjrkrlrmrnrorpdecoder_outputss r(forwardPLBartModel.forwardfs0P  $)>)F 29kk>V>V W   "/3||0#-+0 0O O_==-"1!"4474H14Loa0RV14_1E1I?1-tO ,,  '1"1!"4#1+/    "-??+;;"1"?"?.99,==&5&G&G"1"?"?.99  r')r[rZrY) NNNNNNNNN)r"r#r$r%_tied_weights_keysrrRr`rerrrtorch LongTensorTensorlist FloatTensorrboolr rtupler rr&rErFs@r(rNrNKsP(7'6 | 0  .226596::>(,26:>!%J ##d*J ((4/J !++d2 J !& t 3 J e//047 J J ((4/J  %0047J $;J +,J  u|| 1 1J  J r'rNzv The PLBART Model with a language modeling head. Can be used for code-to-text, text-to-code and code-to-code. ) custom_introc.^\rSrSrSrS/rSS0rS\4U4SjjrSS \ S \ S-S \ S \ R4U4S jjjr S \ S S4Sjr\\\SS\R&S-S\R&S-S\R&S-S\R(S-S\\R,S-S\S-S\R,S-S\R,S-S\R(S-S\ S-S\\S \\R(\-4Sjj555rS\R(4SjrSrU=r$)r6r.r9zlm_head.weightzmodel.shared.weightr-cv>[TU]U5 [U5UlUR S[ R "SURRR455 [R"URURRRSS9Ul UR5 g)Nr9rF)bias)r3rRrNr.register_bufferrzerosrYnum_embeddingsrLinearrXlm_headr\)r:r-r<s r(rR'PLBartForConditionalGeneration.__init__s~   (  0%++q$**BSBSBbBb>c2deyy1B1B1Q1QX]^  r'Nnew_num_tokenspad_to_multiple_of mean_resizingrqcx>[TU]XU5nURURRS5 U$)Nr)r3resize_token_embeddings_resize_final_logits_biasweightshape)r:rrrnew_embeddingsr<s r(r6PLBartForConditionalGeneration.resize_token_embeddingss<8]jk &&~'<'<'B'B1'EFr'c,URRSnX::aURSS2SU24nON[R"SX- 4URRS9n[R "URU/SS9nUR SU5 g)Nr)device)dimr9)r9rrrrcatr)r:rold_num_tokensnew_bias extra_biass r(r8PLBartForConditionalGeneration._resize_final_logits_biass//55b9  +--a..@AHa)H%IRVRhRhRoRopJyy$"8"8*!E1MH 0(;r'rgrhrirjrkrlrmrnlabelsrorpc RU b%Uc"Uc[XRR5nUR"U4UUUUUUUU S.U D6n UR U R 5n XR RU R5-n SnU bF[5nU"U RSURR5U RS55n[UU U RU RU RU R U R"U R$U R&S9 $)a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`] or [`PLBartMultiTokenizer`] depending on the checkpoint. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) PLBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 50003 for *en_XX*, and 50001 for *java*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (: obj:*torch.LongTensor* of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example Mask-filling: ```python >>> from transformers import AutoTokenizer, PLBartForConditionalGeneration >>> model = PLBartForConditionalGeneration.from_pretrained("uclanlp/plbart-base") >>> tokenizer = AutoTokenizer.from_pretrained("uclanlp/plbart-base") >>> # en_XX is the language symbol id for English >>> TXT = " Is 0 the Fibonacci number ? en_XX" >>> input_ids = tokenizer([TXT], add_special_tokens=False, return_tensors="pt").input_ids >>> logits = model(input_ids).logits >>> masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item() >>> probs = logits[0, masked_index].softmax(dim=0) >>> values, predictions = probs.topk(5) >>> tokenizer.decode(predictions).split() ['first', 'same', 'highest', 'result', 'number'] ``` N)rhrirkrjrlrmrnror) losslogitsrlrxryrzr{rvr|)rr-rSr.rrsr9torrviewrTr rlrxryrzr{rvr|)r:rgrhrirjrkrlrmrnrrorpoutputs lm_logitsmasked_lm_lossloss_fcts r(r&PLBartForConditionalGeneration.forwards,@   (-B-J$6v{{?W?W$X!&*jj  ' )/+#9+'"7 '  ' LL!:!:;  6 6 9 9):J:J KK   ')H%innR9O9O&PRXR]R]^`RabN#33")"?"?&99$55&-&G&G")"?"?&99  r'c@[XRR5$r2)rr-rS)r:rs r(%prepare_decoder_input_ids_from_labelsDPLBartForConditionalGeneration.prepare_decoder_input_ids_from_labels>s!&++*B*BCCr')rr.)NT) NNNNNNNNNN) r"r#r$r%r?_keys_to_ignore_on_load_missingrrrRintrr Embeddingrrrrrrrrrrrr rrr rrr&rErFs@r(r6r6s  ':&;#/|ae!7:TzY] <<< .226596::>(,26:>&*!%_ ##d*_ ((4/_ !++d2 _ !& t 3 _ e//047 _ _ ((4/_  %0047_  t#_ $;_ +,_  u||  ._  _ BDELLDDr'r6c\rSrSrSrg)PLBartClassificationHeadiBr Nr!r r'r(rrBr)r'rc(^\rSrSrU4SjrSrU=r$)PLBartForSequenceClassificationiFc :>[5R"S0UD6 g)a  decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`] or [`PLBartMultiTokenizer`] depending on the checkpoint. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) PLBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 50003 for *en_XX*, and 50001 for *java*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (: obj:*torch.LongTensor* of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nr r3r super_kwargsr<s r(r'PLBartForSequenceClassification.forwardGs4 ','r'r )r"r#r$r%rr&rErFs@r(rrFs ((r'rc<^\rSrSr\\U4Sj55rSrU=r$)PLBartForCausalLMidc :>[5R"S0UD6 g)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, PLBartForCausalLM >>> tokenizer = AutoTokenizer.from_pretrained("uclanlp/plbart-base") >>> model = PLBartForCausalLM.from_pretrained("uclanlp/plbart-base") >>> assert model.config.is_decoder, f"{model.__class__} has to be configured as a decoder." >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> expected_shape = [1, inputs.input_ids.shape[-1], model.config.vocab_size] >>> list(logits.shape) == expected_shape True ```Nr rrs r(rPLBartForCausalLM.forwardes2 ','r'r ) r"r#r$r%rrrr&rErFs@r(rrds((r'r)rr6rrNr+)3__doc__rVrrtorch.nnrrr7 cache_utilsr generationrmodeling_outputsr r r modeling_utilsr processing_utilsr utilsrrr utils.genericrutils.output_capturingrbart.modeling_bartrrrrr(bigbird_pegasus.modeling_bigbird_pegasusrmbart.modeling_mbartrconfiguration_plbartrrr+rHrKrNr6rrr__all__r r'r(rs %& ) .&II75_5. 7  2O 2 2 K  K g 'g g T DD%:ODD  DDN 5 (&M(<((: r'