ZjYSrSSKJr SSKJr SSKrSSKJr SSKJrJ r SSK J r SS K J r JrJrJrJr SS KJr SS KJr SS KJr SS KJrJr SSKJr SSKJr \R>"\ 5r!\"SS9\"SS\555r"\\"SS\555r#\"SS9\"SS\555r$\"SS\$55r%\"SS9"SS\$55r&\"SS9"S S!\$\55r'/S"Qr(g)#zRAG model implementation.)Callable) dataclassN)nn)CacheEncoderDecoderCache)PreTrainedConfig)GenerationConfigGenerationMixinGenerationModeLogitsProcessorListStoppingCriteriaList)GENERATION_MODES_MAPPING) ModelOutput)PreTrainedModel)auto_docstringlogging) RagConfig) RagRetrieverzI Base class for retriever augmented marginalized models outputs. ) custom_introc\rSrSr%SrSr\RS-\S'Sr \RS-\S'Sr \RS-\S'Sr \ S-\S'Sr \RS-\S'Sr\RS-\S 'Sr\RS-\S 'Sr\RS-\S 'Sr\RS-\S 'Sr\\RS 4S-\S'Sr\\RS 4S-\S'Sr\RS-\S'Sr\\RS 4S-\S'Sr\\RS 4S-\S'Sr\\RS 4S-\S'Sr\\RS 4S-\S'Sr\\RS 4S-\S'Srg)RetrievAugLMMarginOutput$a loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head. The score is possibly marginalized over all documents for each vocabulary token. doc_scores (`torch.FloatTensor` of shape `(batch_size, config.n_docs)`): Score between each retrieved document embeddings (see `retrieved_doc_embeds`) and `question_encoder_last_hidden_state`. past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains precomputed hidden-states (key and values in the attention blocks) of the decoder that can be used (see `past_key_values` input) to speed up sequential decoding. retrieved_doc_embeds (`torch.FloatTensor` of shape `(batch_size, config.n_docs, hidden_size)`, *optional*, returned when *output_retrieved=True*): Embedded documents retrieved by the retriever. Is used with `question_encoder_last_hidden_state` to compute the `doc_scores`. retrieved_doc_ids (`torch.LongTensor` of shape `(batch_size, config.n_docs)`, *optional*, returned when *output_retrieved=True*): The indexes of the embedded documents retrieved by the retriever. context_input_ids (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Input ids post-processed from the retrieved documents and the question encoder input_ids by the retriever. context_attention_mask (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Attention mask post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. question_encoder_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden states at the output of the last layer of the question encoder pooled output of the model. question_enc_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden states of the question encoder at the output of each layer plus the initial embedding outputs. question_enc_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the question encoder, after the attention softmax, used to compute the weighted average in the self-attention heads. generator_enc_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the generator encoder of the model. generator_enc_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden states of the generator encoder at the output of each layer plus the initial embedding outputs. generator_enc_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the generator encoder, after the attention softmax, used to compute the weighted average in the self-attention heads. generator_dec_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden states of the generator decoder at the output of each layer plus the initial embedding outputs. generator_dec_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the generator decoder, after the attention softmax, used to compute the weighted average in the self-attention heads. generator_cross_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Cross-attentions weights of the generator decoder, after the attention softmax, used to compute the weighted average in the cross-attention heads. Nlosslogits doc_scorespast_key_valuesretrieved_doc_embedsretrieved_doc_idscontext_input_idscontext_attention_mask"question_encoder_last_hidden_state.question_enc_hidden_statesquestion_enc_attentionsgenerator_enc_last_hidden_stategenerator_enc_hidden_statesgenerator_enc_attentionsgenerator_dec_hidden_statesgenerator_dec_attentionsgenerator_cross_attentions)__name__ __module__ __qualname____firstlineno____doc__rtorch FloatTensor__annotations__rrrrrr LongTensorr!r"r#r$tupler%r&r'r(r)r*r+__static_attributes__r,u/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/rag/modeling_rag.pyrr$sDL&*D%  d ")'+FE   $++/J!!D(/$(OUT\(59%++d2915u''$.515u''$.56:E,,t3:CG&(9(9D(@GGKe&7&7&< = DKDHU5#4#4c#9:TAH@D#U%6%6%=DHLu'8'8#'=!>!ELEIeE$5$5s$:;dBIHLu'8'8#'=!>!ELEIeE$5$5s$:;dBIGKe&7&7&< = DKr8rc\rSrSr%SrSr\RS-\S'Sr \RS-\S'Sr \ S-\S'Sr \RS-\S'Sr \RS-\S'Sr\RS-\S 'Sr\RS-\S 'Sr\RS-\S 'Sr\\RS 4S-\S 'Sr\\RS 4S-\S'Sr\RS-\S'Sr\\RS 4S-\S'Sr\\RS 4S-\S'Sr\\RS 4S-\S'Sr\\RS 4S-\S'Sr\\RS 4S-\S'Srg)RetrievAugLMOutputa: logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head. The score is possibly marginalized over all documents for each vocabulary token. doc_scores (`torch.FloatTensor` of shape `(batch_size, config.n_docs)`): Score between each retrieved document embeddings (see `retrieved_doc_embeds`) and `question_encoder_last_hidden_state`. past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains precomputed hidden-states (key and values in the attention blocks) of the decoder that can be used (see `past_key_values` input) to speed up sequential decoding. retrieved_doc_embeds (`torch.FloatTensor` of shape `(batch_size, config.n_docs, hidden_size)`, *optional*, returned when *output_retrieved=True*): Embedded documents retrieved by the retriever. Is used with `question_encoder_last_hidden_state` to compute the `doc_scores`. retrieved_doc_ids (`torch.LongTensor` of shape `(batch_size, config.n_docs)`, *optional*, returned when *output_retrieved=True*): The indexes of the embedded documents retrieved by the retriever. context_input_ids (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Input ids post-processed from the retrieved documents and the question encoder input_ids by the retriever. context_attention_mask (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Attention mask post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. question_encoder_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden states at the output of the last layer of the question encoder pooled output of the model. question_enc_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden states of the question encoder at the output of each layer plus the initial embedding outputs. question_enc_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the question encoder, after the attention softmax, used to compute the weighted average in the self-attention heads. generator_enc_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the generator encoder of the model. generator_enc_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden states of the generator encoder at the output of each layer plus the initial embedding outputs. generator_enc_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the generator encoder, after the attention softmax, used to compute the weighted average in the self-attention heads. generator_dec_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden states of the generator decoder at the output of each layer plus the initial embedding outputs. generator_dec_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the generator decoder, after the attention softmax, used to compute the weighted average in the self-attention heads. generator_cross_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Cross-attentions weights of the generator decoder, after the attention softmax, used to compute the weighted average in the cross-attention heads. Nrrrrr r!r"r#.r$r%r&r'r(r)r*r+r,)r-r.r/r0r1rr2r3r4rrrrr r5r!r"r#r$r6r%r&r'r(r)r*r+r7r,r8r9r;r;sBH(,FE   $++/J!!D(/$(OUT\(59%++d2915u''$.515u''$.56:E,,t3:CG&(9(9D(@GGKe&7&7&< = DKDHU5#4#4c#9:TAH@D#U%6%6%=DHLu'8'8#'=!>!ELEIeE$5$5s$:;dBIHLu'8'8#'=!>!ELEIeE$5$5s$:;dBIGKe&7&7&< = DKr8r;a RAG models were released with the paper [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://huggingface.co/papers/2005.11401) by Patrick Lewis, Ethan Perez, Aleksandra Piktus et al. RAG is a retriever augmented model and encapsulate three components: a question encoder, a dataset retriever and a generator, the encoder and generator are trainable while the retriever is just an indexed dataset. c l\rSrSr%\\S'SrSrSr\ S S\ S-S\ S-S\ S-S \ 4S jj5r S rg) RagPreTrainedModelconfigragTN.question_encoder_pretrained_model_name_or_path'generator_pretrained_model_name_or_path retrieverreturnc UR5VVs0sH,upVURS5(dMU[S5SU_M. nnnUR5VVs0sH,upVURS5(dMU[S5SU_M. nnnUH n USU - M UH n USU - M URSS5n U cLUcS5eSSKJn SU;a#SS KJn U R"U40UDS S 0D6upXS'U R"U40UD6n URSS5nUcNUcS 5eSS KJ n SU;a%SS KJn U R"U40UDS S 0D6unnUUS'UR"U40UD6nURS5nUc,[R"U RUR40UD6nU"XUUS9$s snnfs snnf)a Instantiates an question encoder and a generator from one or two base classes of the library from pretrained model checkpoints. The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated). To train the model, you need to first set it back in training mode with `model.train()`. Params: question_encoder_pretrained_model_name_or_path (`str`, *optional*, defaults to `None`): Information necessary to initiate the question encoder. Can be either: - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co. - A path to a *directory* containing model weights saved using [`~PreTrainedModel.save_pretrained`], e.g., `./my_model_directory/`. generator_pretrained_model_name_or_path (`str`, *optional*, defaults to `None`): Information necessary to initiate the generator. Can be either: - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co. - A path to a *directory* containing model weights saved using [`~PreTrainedModel.save_pretrained`], e.g., `./my_model_directory/`. model_args (remaining positional arguments, *optional*): All remaining positional arguments will be passed to the underlying model's `__init__` method. retriever ([`RagRetriever`], *optional*): The retriever to use. kwwargs (remaining dictionary of keyword arguments, *optional*): Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., `output_attentions=True`). - To update the question_encoder configuration, use the prefix *question_encoder_* for each configuration parameter. - To update the generator configuration, use the prefix *generator_* for each configuration parameter. - To update the parent model configuration, do not use a prefix for each configuration parameter. Behaves differently depending on whether a `config` is provided or automatically loaded. Example: ```python >>> from transformers import RagModel >>> # initialize a RAG from two pretrained models. >>> model = RagModel.from_pretrained_question_encoder_generator( ... "facebook/dpr-question_encoder-single-nq-base", "google-t5/t5-small" ... ) >>> # saving model after fine-tuning >>> model.save_pretrained("./rag") >>> # load fine-tuned model >>> model = RagModel.from_pretrained("./rag") ```question_encoder_N generator_modelznIf `model` is not defined as an argument, a `question_encoder_pretrained_model_name_or_path` has to be defined AutoModelr@) AutoConfigreturn_unused_kwargsTzqIf `generator_model` is not defined as an argument, a `generator_pretrained_model_name_or_path` has to be definedAutoModelForSeq2SeqLM)question_encoder generatorr@rD)items startswithlenpopauto.modeling_autorLauto.configuration_autorMfrom_pretrainedrPgetr'from_question_encoder_generator_configsr@)clsrBrCrDkwargsargumentvaluekwargs_question_encoderkwargs_generatorkeyrQrLrMquestion_encoder_configrRrPgenerator_configr@s r9*from_pretrained_question_encoder_generator=RagPreTrainedModel.from_pretrained_question_encoder_generatorsX|$*<<># #1""#67 8HS,-/ 0% 7#1 # $*<<> #1""<0 1HS&( )5 0#1  +C*S01+#C|c)*$ 366wE  #AM  M 766@CMC]C]BD-D*.D@' 5L1(88> BY  %(($7  :F ! F C//@5?5O5O;6?O6fj62 "2.> *-==7;KI H% >FF '')9)9=CF$4RXdmnnO#  sF=F=G=Gr,)NNN)r-r.r/r0rr4base_model_prefix_supports_flash_attn_supports_sdpa classmethodstrrrrer7r,r8r9r>r>sy NFJ>B)- Bo8;d Bo25tBo $& Bo  BoBor8r>c!^\rSrSrSS\S-S\S-S\S-S\S-4U4Sjjjr\SS\ RS-S \ RS-S \ \ \ RS-S \ RS-S \ RS-S \S-S\ RS-S\ RS-S\ RS-S\S-S\S-S\S-S\S-S\S-S\ \ R\-4Sjj5rSrU=r$)RagModelisNr@rQrRrDc >Uc UbUcS5eUc-[R"URUR40UD6nO1[XR5(dSUSUR35e[ TU]U5 Uc!SSKJn URUR5nUc!SSKJ n URUR5nX@l URb9[U[5(dS[UR5S 35eX@l X l X0l SUlS UlUR%5 g)  question_encoder (`PreTrainedModel`, *optional*): The model responsible for encoding the question into hidden states for retrieval. generator (`PreTrainedModel`, *optional*): The model responsible for generating text based on retrieved documents. retriever (`RagRetriever`, *optional*): The component responsible for retrieving documents from a knowledge base given the encoded question. NzQEither a configuration or an question_encoder and a generator has to be provided.zconfig: z has to be of type rJrKrOz`self.retriever` is of type z&, but should be of type `RagRetriever`F)rr[r@ isinstance config_classsuper__init__rWrL from_configrQrPrRrDrtype ctx_encodercontext_encoder_training post_init) selfr@rQrRrDr]rLrP __class__s r9rsRagModel.__init__usB !&6&ByG\ _ ] >FF '')9)9=CFf&7&788 sHVHL_`d`q`q_r:s s8   # 6(44V5L5LM    B-99&:J:JKI" >> %i66 .tDNN/C.DDjk 6'N 0"(-% r8 input_idsattention_maskencoder_outputsdecoder_input_idsdecoder_attention_maskrrr!r" use_cacheoutput_attentionsoutput_hidden_statesoutput_retrievedn_docsrEc dUbUOURRnU bU OURRn U bU OURRn U bU OURRn U bU OURR n UR SL=(a% USL=(d U SL=(d USL=(a USLnUGc+U(GaURXSS9nUSnUR UUR5RS[RS9R5[URRSS5USS 9nUR(aUS US US US USUS4unn nnnnURU5nU RU5n URU5nURU5nUR!UUSS9R"nUR%SUUR&S5n[R("UR+S5UR-SS55R/S5nOUS US US US4upnnURU5nURU5nU RU5n [R("UR+S5UR-SS55R/S5nOUcS5eU cS5eUcS5eUcS5eUR&SU-S:XdSUSUR&SS35eUbUR1USS9nUbUR1USS9nURUU UUUUU U SS9 nU(d SnSnSnSnSnOWR2nUR4nU(aU (dSnSn SnSn[7S*0SUR8_SU_SUR:_S U_S U _S W_S W_S!W_S"U_S#U_S$UR<_S%UR>_S&UR@_S'URB_S(URD_S)URF_6$)+a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [`RagConfig`], used to initialize the model, specifies which generator to use, it also specifies a compatible generator tokenizer. Use that tokenizer class to obtain the indices. [What are input IDs?](../glossary#input-ids) encoder_outputs (`tuple(tuple(torch.FloatTensor)`, *optional*) Tuple consists of (`generator_enc_last_hidden_state`, *optional*: `generator_enc_hidden_states`, *optional*: `generator_enc_attentions`). `generator_enc_last_hidden_state` of shape `(batch_size, n_docs * sequence_length, hidden_size)` is a sequence of hidden-states at the output of the last layer of the generator's encoder. Used by the ([`RagModel`]) model during decoding. decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Provide for generation tasks. `None` by default, construct as per instructions for the generator model you're using with your RAG instance. decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. doc_scores (`torch.FloatTensor` of shape `(batch_size, config.n_docs)`): Score between each retrieved document embeddings (see `retrieved_doc_embeds`) and `question_encoder_last_hidden_state`. If the model has is not initialized with a `retriever` `doc_scores` has to be provided to the forward pass. `doc_scores` can be computed via `question_encoder_last_hidden_state` and `retrieved_doc_embeds`, see examples for more information. context_input_ids (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Input IDs post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model was not initialized with a `retriever` ``context_input_ids` has to be provided to the forward pass. `context_input_ids` are returned by [`~RagRetriever.__call__`]. context_attention_mask (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`,*optional*, returned when *output_retrieved=True*): Attention mask post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model has is not initialized with a `retriever` `context_attention_mask` has to be provided to the forward pass. `context_attention_mask` are returned by [`~RagRetriever.__call__`]. output_retrieved (`bool`, *optional*): Whether or not to return the `retrieved_doc_embeds`, `retrieved_doc_ids`, `context_input_ids` and `context_attention_mask`. See returned tensors for more detail. n_docs (`int`, *optional*): The number of documents to retrieve. Example: ```python >>> from transformers import AutoTokenizer, RagRetriever, RagModel >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("facebook/rag-token-base") >>> retriever = RagRetriever.from_pretrained( ... "facebook/rag-token-base", index_name="exact", use_dummy_dataset=True ... ) >>> # initialize with RagRetriever to do everything in one forward call >>> model = RagModel.from_pretrained("facebook/rag-token-base", retriever=retriever) >>> inputs = tokenizer("How many people live in Paris?", return_tensors="pt") >>> outputs = model(input_ids=inputs["input_ids"]) ```NT)r} return_dictrcpudevicedtypeprefixptrrreturn_tensorsr!r"rtokenized_doc_idstokenized_doc_attention_maskdoc_idsrrJzMake sure that `context_input_ids` are passed, if no `retriever` is set. Alternatively, you can set a retriever using the `set_retriever(...)` function.zMake sure that `context_attention_mask` are passed, if no `retriever` is set. Alternatively, you can set a retriever using the `set_retriever(...)` function.zMake sure that `doc_scores` are passed, if no `retriever` is set. Alternatively, you can set a retriever using the `set_retriever(...)` function.z^Make sure that `doc_scores` are passed when passing `encoder_outputs` to the forward function.M The first dimension of `context_input_ids` should be a multiple of `n_docs`= , but is .dim) r|r}r~rrrrrrNrrrr r#r$r%r&r'r(r)r*r+r,)$r@rrrrrrDrQdetachtor2float32numpygetattrrRrwrv pooler_outputviewshapebmm unsqueeze transposesqueezerepeat_interleave hidden_states attentionsr;rrencoder_last_hidden_stateencoder_hidden_statesencoder_attentionsdecoder_hidden_statesdecoder_attentionscross_attentions)ryr|r}r~rrrrr!r"rrrrrr]has_to_retrievequestion_enc_outputsr#retriever_outputsrretrieved_doc_input_idsretrieved_doc_attention_maskr gen_outputsr$r%s r9forwardRagModel.forwardsT"-4;;3E3E!*!6IDKK>#8#8(DI!#' %3%!00**=>)*BC)*@A)*=>)*HI))4 ).,/4)):(<()*BC)*@A))4 jf%?SUf,@+B+BCe+f((9(<((,/359:>!%)-,0(,!e ##d*e  t+e uU%6%6784? e !++d2 e !& 0 04 7 e e %%,e !++d2e !& 0 04 7e $;e  $;e #Tke +e d e " u|| 1 1#e e r8rmzu A RAG-sequence model implementation. It performs RAG-sequence specific marginalization in the forward pass. c&^\rSrSrS(S\S-S\S-S\S-S\S-4U4SjjjrS\4SjrS \4S jr \ S)S \ RS-S \ RS-S \\\ RS-S\ RS-S\ RS-S\S-S\ RS-S\ RS-S\ R"S-S\S-S\S-S\S-S\S-S\S-S\S-S\ RS-S\S-S\4$Sjj5r\S5r\S5r\S 5r\ R4"5S*S \ RS-S \ RS-S\ RS-S\ RS-S\ R"S-S!\S-S"\S-S#\S-S\S-S\ R4S$jj5rS+S%jr\S&5rS'rU=r $),RagSequenceForGenerationiNr@rQrRrDc >Uc UbUcS5eUc,[R"URUR40UD6n[TU]U5 [ XX4S9UlUR5 groNzHEither a configuration or an encoder and a generator has to be provided.)r@rQrRrDrr[r@rrrsrmrArxryr@rQrRrDr]rzs r9rs!RagSequenceForGeneration.__init__s !&6&ByG\ V ] >FF '')9)9=CF  6Xaw r8c$XRlgrrArDryrDs r9 set_retriever&RagSequenceForGeneration.set_retriever &r8rvcFSURlXRlgNTrArwrvryrvs r9 set_context_encoder_for_training9RagSequenceForGeneration.set_context_encoder_for_training,0)*r8r|r}r~rrrr!r"rrrrrexclude_bos_score reduce_losslabelsrrEc ,UbUOURRnUbUOURRnUbUOURRnUbUcUnSn UR UUUUUUUU UU U U U US9nSnUb=UR UR URUUURRUUS9n[S0SU_SUR _SUR_SUR_S UR_S UR_S UR_S UR_S UR_SUR _SUR"_SUR$_SUR&_SUR(_SUR*_SUR,_SUR._6$)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [`RagConfig`], used to initialize the model, specifies which generator to use, it also specifies a compatible generator tokenizer. Use that tokenizer class to obtain the indices. [What are input IDs?](../glossary#input-ids) encoder_outputs (`tuple(tuple(torch.FloatTensor)`, *optional*) Tuple consists of (`generator_enc_last_hidden_state`, *optional*: `generator_enc_hidden_states`, *optional*: `generator_enc_attentions`). `generator_enc_last_hidden_state` of shape `(batch_size, n_docs * sequence_length, hidden_size)` is a sequence of hidden-states at the output of the last layer of the generator's encoder. Used by the ([`RagModel`]) model during decoding. decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Provide for generation tasks. `None` by default, construct as per instructions for the generator model you're using with your RAG instance. decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. context_input_ids (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Input IDs post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model was not initialized with a `retriever` ``context_input_ids` has to be provided to the forward pass. `context_input_ids` are returned by [`~RagRetriever.__call__`]. context_attention_mask (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`,*optional*, returned when *output_retrieved=True*): Attention mask post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model has is not initialized with a `retriever` `context_attention_mask` has to be provided to the forward pass. `context_attention_mask` are returned by [`~RagRetriever.__call__`]. doc_scores (`torch.FloatTensor` of shape `(batch_size, config.n_docs)`): Score between each retrieved document embeddings (see `retrieved_doc_embeds`) and `question_encoder_last_hidden_state`. If the model has is not initialized with a `retriever` `doc_scores` has to be provided to the forward pass. `doc_scores` can be computed via `question_encoder_last_hidden_state` and `retrieved_doc_embeds`, see examples for more information. output_retrieved (`bool`, *optional*): Whether or not to return the `retrieved_doc_embeds`, `retrieved_doc_ids`, `context_input_ids` and `context_attention_mask`. See returned tensors for more detail. exclude_bos_score (`bool`, *optional*): Only relevant if `labels` is passed. If `True`, the score of the BOS token is disregarded when computing the loss. reduce_loss (`bool`, *optional*): Only relevant if `labels` is passed. If `True`, the NLL loss is reduced using the `torch.Tensor.sum` operation. n_docs (`int`, *optional*): The number of documents to retrieve. Example: ```python >>> from transformers import AutoTokenizer, RagRetriever, RagSequenceForGeneration >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("facebook/rag-sequence-nq") >>> retriever = RagRetriever.from_pretrained( ... "facebook/rag-sequence-nq", index_name="exact", use_dummy_dataset=True ... ) >>> # initialize with RagRetriever to do everything in one forward call >>> model = RagSequenceForGeneration.from_pretrained("facebook/rag-token-nq", retriever=retriever) >>> inputs = tokenizer("How many people live in Paris?", return_tensors="pt") >>> targets = tokenizer(text_target="In Paris, there are 10 million people.", return_tensors="pt") >>> input_ids = inputs["input_ids"] >>> labels = targets["input_ids"] >>> outputs = model(input_ids=input_ids, labels=labels) >>> # or use retriever separately >>> model = RagSequenceForGeneration.from_pretrained("facebook/rag-sequence-nq", use_dummy_dataset=True) >>> # 1. Encode >>> question_hidden_states = model.question_encoder(input_ids)[0] >>> # 2. Retrieve >>> docs_dict = retriever(input_ids.numpy(), question_hidden_states.detach().numpy(), return_tensors="pt") >>> doc_scores = torch.bmm( ... question_hidden_states.unsqueeze(1), docs_dict["retrieved_doc_embeds"].float().transpose(1, 2) ... ).squeeze(1) >>> # 3. Forward to generator >>> outputs = model( ... context_input_ids=docs_dict["context_input_ids"], ... context_attention_mask=docs_dict["context_attention_mask"], ... doc_scores=doc_scores, ... decoder_input_ids=labels, ... ) ```NFr|r}r~rrr!r"rrrrrrr)repsilonrrrrrrr!r"rr r#r$r%r&r'r(r)r*r+r,)r@rrrrAget_nllrrlabel_smoothingrrr!r"rr r#r$r%r&r'r(r)r*r+)ryr|r}r~rrrr!r"rrrrrrrrrr]outputsrs r9r RagSequenceForGeneration.forwardsN"-4;;3E3E1B1N-TXT_T_TqTq%0%> )) $33  &77  $+#A#A  ")!=!= &77 07/Y/Y (/'I'I %,$C$C -4,S,S )0(K(K &-%E%E )0(K(K &-%E%E! "(/'I'I#  r8c.URR$rrrys r9rD"RagSequenceForGeneration.retriever_xx!!!r8c.URR$rrArRrs r9rR"RagSequenceForGeneration.generatorcrr8c.URR$rrArQrs r9rQ)RagSequenceForGeneration.question_encodergxx(((r8do_deduplicationnum_return_sequences num_beamsc U bU OURRn UbUOURRnUbUOURRn UbUOURRnUc UcS5eUR bUcUR XS9Sn UR UU R5RS[RS9R5[URRSS5U SS 9S nURU5n/n XS 'XS 'SU S 'UbURSOURSU -n[U5GHWnX?U -US-U -nURR "U40U D6nU(aV[R""[%UVs0sHn['UR)55U_M snR+555nURSnUb XUS-R-US5nU"UUSS9nOnUcS5eUcS5eUR-US5nXOU -US-U -nUR-US5nX_US-2SS24nUR-US5nU"UUUUSS9nUS*R/U 5SnU R1UU5 GMZ UR3XRRR4S9$s snf)a Implements RAG sequence "thorough" decoding. Read the [`~generation.GenerationMixin.generate`]` documentation for more information on how to set other generate input parameters. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): The sequence used as a prompt for the generation. If `input_ids` is not passed, then `context_input_ids` has to be provided. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) context_input_ids (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Input IDs post-processed from the retrieved documents and the question encoder input_ids by the retriever. context_attention_mask (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Attention mask post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model is not initialized with a `retriever` or `input_ids` is not given, `context_input_ids` and `context_attention_mask` have to be provided to the forward pass. They are returned by [`~RagRetriever.__call__`]. doc_scores (`torch.FloatTensor` of shape `(batch_size, config.n_docs)`): Score between each retrieved document embeddings (see `retrieved_doc_embeds`) and `question_encoder_last_hidden_state`. If the model is not initialized with a `retriever` or `input_ids` is not given, `doc_scores` has to be provided to the forward pass. `doc_scores` are returned by [`~RagRetriever.__call__`]. do_deduplication (`bool`, *optional*): Whether or not to deduplicate the generations from different context documents for a given input. Has to be set to `False` if used while training with distributed backend. num_return_sequences(`int`, *optional*, defaults to 1): The number of independently computed returned sequences for each element in the batch. Note that this is not the value we pass to the `generator`'s `[`~generation.GenerationMixin.generate`]` function, where we set `num_return_sequences` to `num_beams`. num_beams (`int`, *optional*, defaults to 1): Number of beams for beam search. 1 means no beam search. n_docs (`int`, *optional*, defaults to `config.n_docs`) Number of documents to retrieve and/or number of documents for which to generate an answer. kwargs (`dict[str, Any]`, *optional*): Additional kwargs will be passed to [`~generation.GenerationMixin.generate`]. Return: `torch.LongTensor` of shape `(batch_size * num_return_sequences, sequence_length)`: The generated sequences. The second dimension (sequence length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. Nz= At least one of input_ids or context_input_ids must be givenr}rrrrrrr!rrr}rT)rrzMake sure that `context_attention_mask` are passed, if no `input_ids` is set. Alternatively, you can set a retriever using the `set_retriever(...)` function.zMake sure that `doc_scores` are passed, if no `input_ids` is set. Alternatively, you can set a retriever using the `set_retriever(...)` function.)r!r"rrrr) pad_token_id)r@rrrrrDrQrrr2rrrrRrrangegeneratestacklistrktolistvaluesrepeattopkappend _cat_and_padr)ryr|r}r!r"rrrrr model_kwargsnum_doc_return_sequencesquestion_hidden_stateshypos batch_sizeindexgenerator_input_idsoutput_sequencesknum_candidates new_input_idsrindividual_input_idsindividual_attention_maskindividual_doc_scores top_cand_indss r9r!RagSequenceForGeneration.generateksiB"-4;;3E3E/?/K+QUQ\Q\QmQm$8$D $++JjJj !"+!6IDKK> %*;*C%)%:%:9%:%def%g " $&--/22%u}}2U[[]t~~44hE# !/! " !# !2 4 4Y ? $-[!/8+,)- %&+4+@Y__Q'FWF]F]^_F`djFj :&E"3FNeaiSYEY"Z #~~66#    #(;;tQa4bQaAS_a5GQa4b4i4i4k/l#m -33N $ )%!) < C CNTU V }5EY]^-9T9"-J- (;'A'A"A($-C6>UZ]^U^bhTh,i),E,L,L^]^,_)(2EAI3F3I(J%(=(D(D^UV(W%&:+D4+&* &fo-334LMaPM LL)-8 9g'j  [[5J5J5W5W XXW5cs&$K!c2^^[R"TSS2SS24TRTRSS5R TR R R5/S5mUbUOTR RnTR R=(d TR R RnUSL=(a& TSS2S4RU5R5n UU4Sjn [RRUSS9RURSU-USUR!S55n [RRUSS9R#S5R#S5n U SS2SS2SS2SS24n U SS2SS2SS2SS24nU SS2SS2SS2SS24n[R"XU -U/SS9nTR#S5R#S5R%SUSS5mTR'5UR'5:XdeUR)STS9nUR+SSS 9nU "UU5unnU(a$U (aUSS2SS2SS24R+S5OUR+S5nUR+S5nUR-S5nUR-S5nU*nU*nU(a UR+5nUR+5nUUR!S5- nS U- U-UU--nU$) Nrrc>TRTRRR5nUR 5(a$UR US5 UR US5 UR S5UR S54$Nreqr@rRrany masked_fill_rll smooth_objpad_maskrytargets r9 _mask_pads4RagSequenceForGeneration.get_nll.._mask_padshyy!6!6!C!CDH||~~#.''#6::b>:#5#5b#99 9r8rrrJrrTrkeepdim?)r2catnewrfill_r@rRrr bos_token_idr allr functional log_softmaxrsizerrrgathersum logsumexp)ry seq_logitsrrrrrrruse_bosr seq_logprobs doc_logprobsfirst_token_scoressecond_token_scores remainder rag_logprobsrrnll_loss smooth_losseps_irs` ` r9r RagSequenceForGeneration.get_nlls AqrE]FJJv||A:@@AVAVAcAcd egh "-4;;3E3E{{//U4;;3H3H3U3U d*Rvad||/L/P/P/R :}}000DII   Q 6 )62zr7J }}000CMMbQ[[\^_ *!QA+6*1a1a<8 Aqr1- yy"4L6XZc!djkl !!!$..r299!VQJzz||//1111  Rv 6!%%"d%; #B 3J%6'R1ab\  a rvvay^^A& \\!_))!, 3!k ||~H%//+K,++B//g )EK,?? r8cUSR[SU55[SU555RU5nSnUH:nXBX3URS-2SURS24'X4RS- nM< U$)Nrc3># UHoRSv M g7f)rNr.0ts r9 8RagSequenceForGeneration._cat_and_pad..?s#@1GGAJc3># UHoRSv M g7f)rNr4r5s r9r8r9?sEbZaUVggajZar:r)rr$maxrr)tensorsroutputindr7s r9r%RagSequenceForGeneration._cat_and_pad=s#@#@ @#EbZaEbBbciijvwA;<3qwwqz))'|'+O+.2.2=A59:>(,59:>/3!%)-,0(,)-#'*.!%^ ##d*^  t+^ uU\\23d: ^ !++d2 ^ !& 0 04 7 ^ ^ !++d2^ !& 0 04 7^ %%,^ $;^  $;^ #Tk^ +^  $;^ D[!^ "  4'#^ $d %^ ( ")^ ^ @"""")) ]]_.22659:>/3(,+/ $!TY##d*TY((4/TY!++d2 TY !& 0 04 7 TY %%, TY+TY"DjTY:TYd TY   TYTYnos9vr8rzo A RAG-token model implementation. It performs RAG-token specific marginalization in the forward pass. c&^\rSrSrS0S\S-S\S-S\S-S\S-4U4SjjjrS\4SjrS \4S jr S1S jr \ S 5r \ S 5r \ S5r\S5rS2Sjr\S3S\R(S-S\R*S-S\\\R.S-S\R(S-S\R0S-S\S-S\R(S-S\R(S-S\R*S-S\S-S\S-S\S-S\S-S\S-S\S-S \R(S-S!\S-S"\4$S#jj5r\R<"5SSSSSSSS\"5\ "54 S\R(S-S\R(S-S\R(S-S\R(S-S\R*S-S!\S-S$\!S-S%\"\\R./\#\4S-S&\S-S'\ S-S"\R(4S(jj5r$S)r%S*r&S+r'S,r(S2S-jr)S4S.jr*S/r+U=r,$)5RagTokenForGenerationiGNr@rQrRrDc >Uc UbUcS5eUc,[R"URUR40UD6n[TU]U5 [ XX4S9UlUR5 grrrs r9rsRagTokenForGeneration.__init__Ms !&6&ByG\ V ] >FF '')9)9=CF  6Xaw r8c$XRlgrrrs r9r#RagTokenForGeneration.set_retrievermrr8rvcFSURlXRlgrrrs r9r6RagTokenForGeneration.set_context_encoder_for_trainingprr8c 6Ub USS2SS24nSUUUUUUSUS. $)NrT) r|r~rr"rrrdo_marginalizerr,) ryrrr}rr~rrr]s r9prepare_inputs_for_generation3RagTokenForGeneration.prepare_inputs_for_generationtsB  & 1!RS& 9 .$&4!2.""  r8c.URR$rrrs r9rDRagTokenForGeneration.retrieverrr8c.URR$rrrs r9rRRagTokenForGeneration.generatorrr8c.URR$rrrs r9rQ&RagTokenForGeneration.question_encoderrr8cZ^^ Sm Sn[[U55Hn[U[5(aU U4SjURR UR URR URURR UR URR UR45upEpgXEXg4nOBU U4SjUR UR UR UR45upEXE4nX(4- nM [U5"U5$)zeReorders cache for generation. BART-inspired but we need to take care of the extra dimension for docscURSURS-nUR"SU/URSSQ76nURSU5nUR"S/URSSQ76nU$)NrrrrJ)rr index_select)r new_orderrresults r9_reorder_stacked>RagTokenForGeneration._reorder_cache.._reorder_stackeds"((+yq/AAF)..r6TM# UH'nT"UTRUR55v M) g7frrrr6xr]beam_idxs r9r87RagTokenForGeneration._reorder_cache..s4\%Q AHH(=>>/2c3h># UH'nT"UTRUR55v M) g7frr`ras r9r8rds/6c%Q AHH(=>>cre) rrUrprself_attention_cachelayerskeysrcross_attention_cacheru) rrcreordered_pastidxself_attention_kself_attention_vcross_attention_kcross_attention_v new_tupler]s ` @r9_reorder_cache$RagTokenForGeneration._reorder_caches# _-.C/+>??\(<<CCCHMM'<<CCCHOO'==DDSINN'==DDSIPP \X 4E.ARf 6-44S9>>@V@VWZ@[@b@bc62 .@ l *N%/&O$^44r8cnUbUOURRn[RR USS9R UR SU-USURS55n[R"USS9nXERS5RS5-n[R"USS9$)Nrrrr) r@rrr r!rrr"r2rr%)ryr&rrr(r) log_prob_sums r9 marginalize!RagTokenForGeneration.marginalizes!-4;;3E3E}}000DII   Q 6 )62zr7J ((; #&<&>> from transformers import AutoTokenizer, RagRetriever, RagTokenForGeneration >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("facebook/rag-token-nq") >>> retriever = RagRetriever.from_pretrained( ... "facebook/rag-token-nq", index_name="exact", use_dummy_dataset=True ... ) >>> # initialize with RagRetriever to do everything in one forward call >>> model = RagTokenForGeneration.from_pretrained("facebook/rag-token-nq", retriever=retriever) >>> inputs = tokenizer("How many people live in Paris?", return_tensors="pt") >>> targets = tokenizer(text_target="In Paris, there are 10 million people.", return_tensors="pt") >>> input_ids = inputs["input_ids"] >>> labels = targets["input_ids"] >>> outputs = model(input_ids=input_ids, labels=labels) >>> # or use retriever separately >>> model = RagTokenForGeneration.from_pretrained("facebook/rag-token-nq", use_dummy_dataset=True) >>> # 1. Encode >>> question_hidden_states = model.question_encoder(input_ids)[0] >>> # 2. Retrieve >>> docs_dict = retriever(input_ids.numpy(), question_hidden_states.detach().numpy(), return_tensors="pt") >>> doc_scores = torch.bmm( ... question_hidden_states.unsqueeze(1), docs_dict["retrieved_doc_embeds"].float().transpose(1, 2) ... ).squeeze(1) >>> # 3. Forward to generator >>> outputs = model( ... context_input_ids=docs_dict["context_input_ids"], ... context_attention_mask=docs_dict["context_attention_mask"], ... doc_scores=doc_scores, ... decoder_input_ids=labels, ... ) >>> # or directly generate >>> generated = model.generate( ... context_input_ids=docs_dict["context_input_ids"], ... context_attention_mask=docs_dict["context_attention_mask"], ... doc_scores=doc_scores, ... ) >>> generated_string = tokenizer.batch_decode(generated, skip_special_tokens=True) ```NFr)rrrrrrrr!r"rr r#r$r%r&r'r(r)r*r+r,)r@rrOrrArrrrrvrrr!r"rr r#r$r%r&r'r(r)r*r+)ryr|r}r~rrrr!r"rrrrrrOrrrr]rrrs r9rRagTokenForGeneration.forwards"^"-4;;3E3E+9+E4;;KeKe%0%ST-S:XdSTSUR>SS35eUR>ST-mUR@R4RC5nU"X4SS9n[.RD"TURF-S4URH[.RJ[MURO55RPS9nUR>SnUSnS%UU4SjjnU"XGRFS9nU"UURFS9US'URSURFSS9nX]S'UU S'XMS'TU S'URTU S 'URWUUUUU URPS!9nURYXzS"9nUR[UU SUR>SUR\S- S#9 U"UU4UUUS$.U DU D6$)&a Implements RAG token decoding. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): The sequence used as a prompt for the generation. If `input_ids` is not passed, then `context_input_ids` has to be provided. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) context_input_ids (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Input IDs post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model has is not initialized with a `retriever`, `context_input_ids` has to be provided to the forward pass. `context_input_ids` are returned by [`~RagRetriever.__call__`]. context_attention_mask (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Attention mask post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model has is not initialized with a `retriever`, `context_input_ids` has to be provided to the forward pass. `context_input_ids` are returned by [`~RagRetriever.__call__`]. doc_scores (`torch.FloatTensor` of shape `(batch_size, config.n_docs)`): Score between each retrieved document embeddings (see `retrieved_doc_embeds`) and `question_encoder_last_hidden_state`. If the model has is not initialized with a `retriever`, `context_input_ids` has to be provided to the forward pass. `context_input_ids` are returned by [`~RagRetriever.__call__`]. n_docs (`int`, *optional*, defaults to `config.n_docs`) Number of documents to retrieve and/or number of documents for which to generate an answer. generation_config (`~generation.GenerationConfig`, *optional*): The generation configuration to be used as base parametrization for the generation call. `**kwargs` passed to generate matching the attributes of `generation_config` will override them. If `generation_config` is not provided, the default will be used, which has the following loading priority: 1) from the `generation_config.json` model file, if it exists; 2) from the model configuration. Please note that unspecified parameters will inherit [`~generation.GenerationConfig`]'s default values, whose documentation should be checked to parameterize generation. prefix_allowed_tokens_fn (`Callable[[int, torch.Tensor], list[int]]`, *optional*): If provided, this function constraints the beam search to allowed tokens only at each step. If not provided no constraint is applied. This function takes 2 arguments `inputs_ids` and the batch ID `batch_id`. It has to return a list with the allowed tokens for the next generation step conditioned on the previously generated tokens `inputs_ids` and the batch ID `batch_id`. This argument is useful for constrained generation conditioned on the prefix, as described in [Autoregressive Entity Retrieval](https://huggingface.co/papers/2010.00904). logits_processor (`LogitsProcessorList`, *optional*): Custom logits processors that complement the default logits processors built from arguments and a model's config. If a logit processor is passed that is already created with the arguments or a model's config an error is thrown. stopping_criteria (`StoppingCriteriaList`, *optional*): Custom stopping criteria that complement the default stopping criteria built from arguments and a model's config. If a stopping criteria is passed that is already created with the arguments or a model's config an error is thrown. kwargs (`dict[str, Any]`, *optional*): Ad hoc parametrization of `generate_config` and/or additional model-specific kwargs that will be forwarded to the `forward` function of the model. Return: `torch.LongTensor` of shape `(batch_size * num_return_sequences, sequence_length)`: The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. NFz!RAG model is not compatible with z5 generation. Please check your generation parameters.r}rrrrrrrr!r"rrrJrrrT)r|r}r)rrrlast_hidden_statec>USSSS24RTST4URSS-5nURTUT4URSS-5nURTU-T-4URSS-5$)Nrr)reshaperexpand)tensorrrrs r9extend_enc_output9RagTokenForGeneration.generate..extend_enc_output sD$M*22J63JV\\Z[Z\M]3]^F]]J 6#BV\\RSRTEU#UVF>>: #9F#B"Dv||TUTVGW"WX Xr8)rrrr~rr)rzinput_ids_seq_lengthencoder_input_idsr{r|r)rzr})generation_modermax_cache_length)r|r}rzr)/_extract_generation_mode_kwargs_prepare_generation_configget_generation_moder SAMPLE GREEDY_SEARCH BEAM_SEARCH BEAM_SAMPLE ValueErrorrrur_validate_model_kwargscopy_validate_generation_moderZ_prepare_special_tokensr@rrDrQrrr2rrrRrrrrrrA get_encoderfullrdecoder_start_token_idlongnext parametersrrr_get_logits_processor_get_stopping_criteria_prepare_cache_for_generation max_length)ryr|r}r!r"rrrzr{r|r}r]generation_mode_kwargsrrdecoding_methodkwargs_has_attention_maskroutrencoderr~rrrprepared_logits_processorprepared_stopping_criteriars ` @r9rRagTokenForGeneration.generatevsyb"&!E!EdFTY[_ae!f*.*I*IJ[*f_e*f'+??A   ! !  ( (  & &  & & #  3O3DDyz "$t*.F.WX ##L$5$5$78 &&Kab$0$4$45Et$LTX$X! $$%68QR"-4;;3E3E >> %*;*C%)%:%:9%:%def%g "..&--/22%u}}2U[[]t~~44hE# !C'(,-*+8L5$8#:#:;Q#R 1 4 4Y ? %;%>%>y%I "#9#C#CA#FH\HfHfghjkHlmuuJ"''*V39 [\b[cd!''*+1 . 9 ',,Q/69 (($$002!,=rvwJJ +55 5q 9  4 4**)*11  )r2+,?@ Y Y"33IUpUp!q/@ ):)D)D0 +, 112C2M2MST1U &0\"*9 &')?%&!' X$5$?$? [!$($>$>/!5/%=-## %?% !&*%@%@/&A& " **    q).99A= +    78/  %    r8c(URX5nU$r)rr)ryrrcs r9_temporary_reorder_cache.RagTokenForGeneration._temporary_reorder_cacheCs--oHr8cJURRR5$r)rArRget_input_embeddingsrs r9r*RagTokenForGeneration.get_input_embeddingsJsxx!!6688r8cJURRR5$r)rArRget_output_embeddingsrs r9r+RagTokenForGeneration.get_output_embeddingsMsxx!!7799r8cLURRRU5$r)rArRset_output_embeddings)rynew_embeddingss r9r+RagTokenForGeneration.set_output_embeddingsPsxx!!77GGr8cUcURRnURUR5nUSS2SS24R 5USS2SS24'X#SS2S4'U$)zCShift input ids one token to the right, and pad with start_token_idNrrr)r@r new_zerosrclone)ryr|start_token_idshifted_input_idss r9shift_tokens_right(RagTokenForGeneration.shift_tokens_rightSse  !![[??N%// @#,QV#4#:#:#<!QR% "0!Q$  r8c^^UbUOTRRn[R"TSS2SS24TR TR SS5R TRRR5/S5mUU4SjnTRXU5nTRS5mTR5UR5:XdeURSTS9n URSSS9n U"X5upU RS5n U RS5n U *n U *n U(a U R5n U R5n XXRS5- n SU- U -X--nU$) Nrrc>TRTRRR5nUR 5(a$UR US5 UR US5 UR S5UR S54$r r rs r9r1RagTokenForGeneration.get_nll.._mask_padscrr8rrTrr)r@rr2rrrrrRrrvrrr#r$r")ryr&rrrrrrr-rrr.r/r0rs` ` r9rRagTokenForGeneration.get_nll\sW!-4;;3E3E AqrE]FJJv||A:@@AVAVAcAcd egh  :'' G !!"%zz||//1111  Rv 6!%%"d%; #B3 VVAY^^A& 3!k ||~H%//+K++B//g )E,?? r8rAr)NNNNNNrrB)Fr N)-r-r.r/r0r rrrsrrrPrCrDrRrQrErrrvrr2r5r3r6rrrrrrrrDr rr rrrrrrrrrr7rrs@r9rGrGGs+/37,0)-  4'*D0#T)   $& @'|'+O+ :""""))55@ 4.237=A59:>(,59:>/3!%)-,0(,&*#'*.!%j ##d*j ))D0j uU\\23d: j !++d2 j !& 0 04 7 j j !++d2j !& 0 04 7j %%,j $;j  $;j #Tkj +j t j D[!j "  4'#j $d %j ( ")j j X ]]_.22659:>/3!59TX7J7L9M9OI ##d*I ((4/I !++d2 I !& 0 04 7 I %%, I d I ,d2I #+C+>S +I"JT"QI .4I 0$6I    I I X9:H!""r8rG)rmr>rrG))r1collections.abcr dataclassesrr2r cache_utilsrrconfiguration_utilsr generationr r r r rgeneration.utilsrmodeling_outputsrmodeling_utilsrutilsrrconfiguration_ragr retrieval_ragr get_loggerr-loggerrr;r>rmrrG__all__r,r8r9rsZ $! 53vv8+-,('   H %  WL{WL  WLt TLTL TLnIoIoIoX[ ![ [ | m1m  m`  r.r  rj br8