U ,-epP@sNdZddlZddlmZddlmZmZmZmZm Z ddl Z ddl m Z ddl m Z ddlmZmZmZmZdd lmZdd lmZdd lmZmZmZd d lmZd dlmZee Z!dZ"eGdddeZ#eGdddeZ$GdddeZ%dZ&dZ'ee&Gddde%Z(ede&Gddde%Z)ede&Gddde%Z*dS) zRAG model implementation.N) dataclass)CallableListOptionalTupleUnion)nn)PretrainedConfig)BeamSearchScorerGenerationConfigLogitsProcessorListStoppingCriteriaList) ModelOutput)PreTrainedModel)%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) RagConfig) RagRetrieverrc@s\eZdZUdZdZeejed<dZ ejed<dZ ejed<dZ ee ejed<dZ eejed<dZeejed<dZeejed <dZeejed <dZeejed <dZeeejed <dZeeejed <dZeejed<dZeeejed<dZeeejed<dZeeejed<dZeeejed<dZeeejed<dS)RetrievAugLMMarginOutputa Base class for retriever augmented marginalized models outputs. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head. The score is possibly marginalized over all documents for each vocabulary token. doc_scores (`torch.FloatTensor` of shape `(batch_size, config.n_docs)`): Score between each retrieved document embeddings (see `retrieved_doc_embeds`) and `question_encoder_last_hidden_state`. past_key_values (`List[torch.FloatTensor]`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): List of `torch.FloatTensor` of length `config.n_layers`, with each tensor of shape `(2, batch_size, num_heads, sequence_length, embed_size_per_head)`). Contains precomputed hidden-states (key and values in the attention blocks) of the decoder that can be used (see `past_key_values` input) to speed up sequential decoding. retrieved_doc_embeds (`torch.FloatTensor` of shape `(batch_size, config.n_docs, hidden_size)`, *optional*, returned when *output_retrieved=True*): Embedded documents retrieved by the retriever. Is used with `question_encoder_last_hidden_state` to compute the `doc_scores`. retrieved_doc_ids (`torch.LongTensor` of shape `(batch_size, config.n_docs)`, *optional*, returned when *output_retrieved=True*): The indexes of the embedded documents retrieved by the retriever. context_input_ids (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Input ids post-processed from the retrieved documents and the question encoder input_ids by the retriever. context_attention_mask (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Attention mask post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. question_encoder_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden states at the output of the last layer of the question encoder pooled output of the model. question_enc_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden states of the question encoder at the output of each layer plus the initial embedding outputs. question_enc_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the question encoder, after the attention softmax, used to compute the weighted average in the self-attention heads. generator_enc_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the generator encoder of the model. generator_enc_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden states of the generator encoder at the output of each layer plus the initial embedding outputs. generator_enc_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the generator encoder, after the attention softmax, used to compute the weighted average in the self-attention heads. generator_dec_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden states of the generator decoder at the output of each layer plus the initial embedding outputs. generator_dec_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the generator decoder, after the attention softmax, used to compute the weighted average in the self-attention heads. generator_cross_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Cross-attentions weights of the generator decoder, after the attention softmax, used to compute the weighted average in the cross-attention heads. Nlosslogits doc_scorespast_key_valuesretrieved_doc_embedsretrieved_doc_idscontext_input_idscontext_attention_mask"question_encoder_last_hidden_statequestion_enc_hidden_statesquestion_enc_attentionsgenerator_enc_last_hidden_stategenerator_enc_hidden_statesgenerator_enc_attentionsgenerator_dec_hidden_statesgenerator_dec_attentionsgenerator_cross_attentions)__name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rrrrrr LongTensorrrr r!rr"r#r$r%r&r'r(r1r1e/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/models/rag/modeling_rag.pyr&s$ Jrc@sJeZdZUdZdZejed<dZejed<dZ e e ejed<dZ e ejed<dZ e ejed<dZe ejed<dZe ejed <dZe ejed <dZe eejed <dZe eejed <dZe ejed <dZe eejed<dZe eejed<dZe eejed<dZe eejed<dZe eejed<dS)RetrievAugLMOutputa; Args: logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head. The score is possibly marginalized over all documents for each vocabulary token. doc_scores (`torch.FloatTensor` of shape `(batch_size, config.n_docs)`): Score between each retrieved document embeddings (see `retrieved_doc_embeds`) and `question_encoder_last_hidden_state`. past_key_values (`List[torch.FloatTensor]`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): List of `torch.FloatTensor` of length `config.n_layers`, with each tensor of shape `(2, batch_size, num_heads, sequence_length, embed_size_per_head)`). Contains precomputed hidden-states (key and values in the attention blocks) of the decoder that can be used (see `past_key_values` input) to speed up sequential decoding. retrieved_doc_embeds (`torch.FloatTensor` of shape `(batch_size, config.n_docs, hidden_size)`, *optional*, returned when *output_retrieved=True*): Embedded documents retrieved by the retriever. Is used with `question_encoder_last_hidden_state` to compute the `doc_scores`. retrieved_doc_ids (`torch.LongTensor` of shape `(batch_size, config.n_docs)`, *optional*, returned when *output_retrieved=True*): The indexes of the embedded documents retrieved by the retriever. context_input_ids (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Input ids post-processed from the retrieved documents and the question encoder input_ids by the retriever. context_attention_mask (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Attention mask post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. question_encoder_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden states at the output of the last layer of the question encoder pooled output of the model. question_enc_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden states of the question encoder at the output of each layer plus the initial embedding outputs. question_enc_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the question encoder, after the attention softmax, used to compute the weighted average in the self-attention heads. generator_enc_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the generator encoder of the model. generator_enc_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden states of the generator encoder at the output of each layer plus the initial embedding outputs. generator_enc_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the generator encoder, after the attention softmax, used to compute the weighted average in the self-attention heads. generator_dec_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden states of the generator decoder at the output of each layer plus the initial embedding outputs. generator_dec_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the generator decoder, after the attention softmax, used to compute the weighted average in the self-attention heads. generator_cross_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Cross-attentions weights of the generator decoder, after the attention softmax, used to compute the weighted average in the cross-attention heads. Nrrrrrrrr r!r"r#r$r%r&r'r()r)r*r+r,rr-r.r/rrrrrrr0rrr r!rr"r#r$r%r&r'r(r1r1r1r2r3s" Fr3csFeZdZdZeZdZefddZed e e e e dddZ Z S) RagPreTrainedModela RAG models were released with the paper [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/abs/2005.11401) by Patrick Lewis, Ethan Perez, Aleksandra Piktus et al. RAG is a retriever augmented model and encapsulate three components: a question encoder, a dataset retriever and a generator, the encoder and generator are trainable while the retriever is just an indexed dataset. ragcsd|d<tj||S)NFZ _fast_init)superfrom_pretrained)clsargskwargs __class__r1r2r7sz"RagPreTrainedModel.from_pretrainedN).question_encoder_pretrained_model_name_or_path'generator_pretrained_model_name_or_path retrieverreturncKszdd|D}dd|D}|D]}|d|=q,|D]}|d|=qD|dd}|dkr|dk sxtdd d lm} d |krd d lm} | j|f|d di\} }| |d <| j|f|}|dd} | dkr@|dk stdd dlm } d |kr2d d lm} | j|f|d di\}}||d <| j|f|} | d d}|dkrjt j |j | j f|}||| ||dS)a5 Instantiates an question encoder and a generator from one or two base classes of the library from pretrained model checkpoints. The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated). To train the model, you need to first set it back in training mode with `model.train()`. Params: question_encoder_pretrained_model_name_or_path (`str`, *optional*, defaults to `None`): Information necessary to initiate the question encoder. Can be either: - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co. Valid model ids can be located at the root-level, like `bert-base-uncased`, or namespaced under a user or organization name, like `dbmdz/bert-base-german-cased`. - A path to a *directory* containing model weights saved using [`~PreTrainedModel.save_pretrained`], e.g., `./my_model_directory/`. - A path or url to a *tensorflow index checkpoint file* (e.g, `./tf_model/model.ckpt.index`). In this case, `from_tf` should be set to `True` and a configuration object should be provided as `config` argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards. generator_pretrained_model_name_or_path (`str`, *optional*, defaults to `None`): Information necessary to initiate the generator. Can be either: - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co. Valid model ids can be located at the root-level, like `bert-base-uncased`, or namespaced under a user or organization name, like `dbmdz/bert-base-german-cased`. - A path to a *directory* containing model weights saved using [`~PreTrainedModel.save_pretrained`], e.g., `./my_model_directory/`. - A path or url to a *tensorflow index checkpoint file* (e.g, `./tf_model/model.ckpt.index`). In this case, `from_tf` should be set to `True` and a configuration object should be provided as `config` argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards. model_args (remaining positional arguments, *optional*): All remaining positional arguments will be passed to the underlying model's `__init__` method. retriever ([`RagRetriever`], *optional*): The retriever to use. kwwargs (remaining dictionary of keyword arguments, *optional*): Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., `output_attentions=True`). - To update the question_encoder configuration, use the prefix *question_encoder_* for each configuration parameter. - To update the generator configuration, use the prefix *generator_* for each configuration parameter. - To update the parent model configuration, do not use a prefix for each configuration parameter. Behaves differently depending on whether a `config` is provided or automatically loaded. Example: ```python >>> from transformers import RagModel >>> # initialize a RAG from two pretrained models. >>> model = RagModel.from_pretrained_question_encoder_generator( ... "facebook/dpr-question_encoder-single-nq-base", "t5-small" ... ) >>> # saving model after fine-tuning >>> model.save_pretrained("./rag") >>> # load fine-tuned model >>> model = RagModel.from_pretrained("./rag") ```cSs,i|]$\}}|dr|tdd|qS)question_encoder_N startswithlen.0argumentvaluer1r1r2 :s zQRagPreTrainedModel.from_pretrained_question_encoder_generator..cSs,i|]$\}}|dr|tdd|qS) generator_NrBrEr1r1r2rI@s rArJmodelNznIf `model` is not defined as an argument, a `question_encoder_pretrained_model_name_or_path` has to be defined AutoModelconfig) AutoConfigZreturn_unused_kwargsTzqIf `generator_model` is not defined as an argument, a `generator_pretrained_model_name_or_path` has to be definedAutoModelForSeq2SeqLM)question_encoder generatorrOr?)itemskeyspopAssertionErrorauto.modeling_autorNZauto.configuration_autorPr7rRgetr'from_question_encoder_generator_configsrO)r8r=r>r?r:Zkwargs_question_encoderZkwargs_generatorkeyrSrNrPZquestion_encoder_configrTrRZgenerator_configrOr1r1r2*from_pretrained_question_encoder_generatorszH                  z=RagPreTrainedModel.from_pretrained_question_encoder_generator)NNN)r)r*r+r,r config_classZbase_model_prefix classmethodr7strrrr] __classcell__r1r1r;r2r4sr4a RAG is a seq2seq model which encapsulates two core components: a question encoder and a generator. During a forward pass, we encode the input with the question encoder and pass it to the retriever to extract relevant context documents. The documents are then prepended to the input. Such contextualized inputs is passed to the generator. The question encoder can be any *autoencoding* model, preferably [`DPRQuestionEncoder`], and the generator can be any *seq2seq* model, preferably [`BartForConditionalGeneration`]. The model can be initialized with a [`RagRetriever`] for end-to-end generation or used in combination with the outputs of a retriever in multiple steps---see examples for more details. The model is compatible any *autoencoding* model as the `question_encoder` and any *seq2seq* model with language model head as the `generator`. It has been tested with [`DPRQuestionEncoder`] as the `question_encoder` and [`BartForConditionalGeneration`] or [`T5ForConditionalGeneration`] as the `generator`. This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Args: config ([`RagConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. question_encoder ([`PreTrainedModel`]): An encoder model compatible with the faiss index encapsulated by the `retriever`. generator ([`PreTrainedModel`]): A seq2seq model used as the generator in the RAG architecture. retriever ([`RagRetriever`]): A retriever class encapsulating a faiss index queried to obtain context documents for current inputs. aO Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [`RagConfig`], used to initialize the model, specifies which generator to use, it also specifies a compatible generator tokenizer. Use that tokenizer class to obtain the indices. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_outputs (`tuple(tuple(torch.FloatTensor)`, *optional*) Tuple consists of (`generator_enc_last_hidden_state`, *optional*: `generator_enc_hidden_states`, *optional*: `generator_enc_attentions`). `generator_enc_last_hidden_state` of shape `(batch_size, n_docs * sequence_length, hidden_size)` is a sequence of hidden-states at the output of the last layer of the generator's encoder. Used by the ([`RagModel`]) model during decoding. decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Provide for generation tasks. `None` by default, construct as per instructions for the generator model you're using with your RAG instance. decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. past_key_values (`tuple(tuple(torch.FloatTensor))`): Tuple consists of two elements: `encoder_outputs` of the RAG model (see `encoder_outputs`) and `past_key_values` of the underlying generator. Can be used to speed up decoding. `past_key_values` are used in the ([`RagTokenForGeneration`]) model during decoding. doc_scores (`torch.FloatTensor` of shape `(batch_size, config.n_docs)`): Score between each retrieved document embeddings (see `retrieved_doc_embeds`) and `question_encoder_last_hidden_state`. If the model has is not initialized with a `retriever` `doc_scores` has to be provided to the forward pass. `doc_scores` can be computed via `question_encoder_last_hidden_state` and `retrieved_doc_embeds`, see examples for more information. context_input_ids (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Input IDs post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model was not initialized with a `retriever` ``context_input_ids` has to be provided to the forward pass. `context_input_ids` are returned by [`~RagRetriever.__call__`]. context_attention_mask (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`,*optional*, returned when *output_retrieved=True*): Attention mask post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model has is not initialized with a `retriever` `context_attention_mask` has to be provided to the forward pass. `context_attention_mask` are returned by [`~RagRetriever.__call__`]. use_cache (`bool`, *optional*, defaults to `True`): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. output_retrieved(`bool`, *optional*): Whether or not to return the `retrieved_doc_embeds`, `retrieved_doc_ids`, `context_input_ids` and `context_attention_mask`. See returned tensors for more detail. n_docs (`int`, *optional*, defaults to `config.n_docs``) Number of documents to retrieve and/or number of documents for which to generate an answer. cseZdZd eeeeeeeedfdd Zee e e e dd ee jee jeeee jee jee jeeee jee jee jee jeeeeeeeeeeeee je fdddZZS) RagModelNrOrSrTr?c s|dk s |dk r|dk s td|dkr>tj|j|jf|}n"t||js`td|d|jt||dkrddlm }| |j }|dkrddlm }| |j }||_|jdk rt|tstdt|jd||_||_ ||_ d|_d |_dS) NzQEither a configuration or an question_encoder and a generator has to be provided.zconfig: z has to be of type rLrMrQz`self.retriever` is of type z&, but should be of type `RagRetriever`F)rXrr[rO isinstancer^r6__init__rYrN from_configrSrRrTr?rtype ctx_encodercontext_encoder_training)selfrOrSrTr?r:rNrRr;r1r2resF"      zRagModel.__init__ output_typer^) input_idsattention_maskencoder_outputsdecoder_input_idsdecoder_attention_maskrrrr use_cacheoutput_attentionsoutput_hidden_statesoutput_retrievedn_docsr@cCs4|dk r |n|jj}| dk r | n|jj} | dk r4| n|jj} | dk rH| n|jj} | dk r\| n|jj} |jdk o|dks| dks|dko|dk}|dkr*|r|j||dd}|d}|j|| t j  |jjj|dd}|jr|d|d|d |d |d |d f\}} }}}}| |}| |} | |}| |}|j||ddj}|d ||jd}t |d|ddd}nb|d|d|d |d f\}} }}| |}| |}| |} t |d|ddd}n6|dk std| dk std|dk s*td|dk s>> from transformers import AutoTokenizer, RagRetriever, RagModel >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("facebook/rag-token-base") >>> retriever = RagRetriever.from_pretrained( ... "facebook/rag-token-base", index_name="exact", use_dummy_dataset=True ... ) >>> # initialize with RagRetriever to do everything in one forward call >>> model = RagModel.from_pretrained("facebook/rag-token-base", retriever=retriever) >>> inputs = tokenizer("How many people live in Paris?", return_tensors="pt") >>> outputs = model(input_ids=inputs["input_ids"]) ```NT)rn return_dictrptprefixrvZreturn_tensorsrrrZtokenized_doc_idsZtokenized_doc_attention_maskZdoc_idsrrLzMake sure that `context_input_ids` are passed, if no `retriever` is set. Alternatively, you can set a retriever using the `set_retriever(...)` function.zMake sure that `context_attention_mask` are passed, if no `retriever` is set. Alternatively, you can set a retriever using the `set_retriever(...)` function.zMake sure that `doc_scores` are passed, if no `retriever` is set. Alternatively, you can set a retriever using the `set_retriever(...)` function.z^Make sure that `doc_scores` are passed when passing `encoder_outputs` to the forward function.M The first dimension of `context_input_ids` should be a multiple of `n_docs`= , but is .dim) rmrnrorprqrrrrsrwN)rrrrrrrr r!r"r#r$r%r&r'r()&rOrvrrrsrtrur?rScpudetachtor-float32numpyrTrzrirhZ pooler_outputviewshapebmm unsqueeze transposesqueezerXrepeat_interleave hidden_statesZ attentionsr3rrZencoder_last_hidden_stateZencoder_hidden_statesZencoder_attentionsZdecoder_hidden_statesZdecoder_attentionsZcross_attentions)rjrmrnrorprqrrrrrrrsrtrurvZhas_to_retrieveZquestion_enc_outputsr Zretriever_outputsrZretrived_doc_input_idsZretrived_doc_attention_maskrZ gen_outputsr!r"r1r1r2forwards&                    zRagModel.forward)NNNN)NNNNNNNNNNNNNN)r)r*r+rr rrrerRAG_FORWARD_INPUTS_DOCSTRINGrr3_CONFIG_FOR_DOCr-r0Tensorrr. BoolTensorboolintrrrar1r1r;r2rbsV* rbzu A RAG-sequence model implementation. It performs RAG-sequence specific marginalization in the forward pass. cseZdZdeeeeeeeedfdd ZedddZedd d Z e e e e ed deejeejeeeejeejeejeeeejeejeejeejeeeeeeeeeeeeeejeee d d dZeddZeddZeddZed eejeejeejeejeejeeeeeeeeejd ddZd!ddZeddZ Z!S)"RagSequenceForGenerationNrcc s^|dk s |dk r|dk s td|dkr>> from transformers import AutoTokenizer, RagRetriever, RagSequenceForGeneration >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("facebook/rag-sequence-nq") >>> retriever = RagRetriever.from_pretrained( ... "facebook/rag-sequence-nq", index_name="exact", use_dummy_dataset=True ... ) >>> # initialize with RagRetriever to do everything in one forward call >>> model = RagSequenceForGeneration.from_pretrained("facebook/rag-token-nq", retriever=retriever) >>> inputs = tokenizer("How many people live in Paris?", return_tensors="pt") >>> targets = tokenizer(text_target="In Paris, there are 10 million people.", return_tensors="pt") >>> input_ids = inputs["input_ids"] >>> labels = targets["input_ids"] >>> outputs = model(input_ids=input_ids, labels=labels) >>> # or use retriever separately >>> model = RagSequenceForGeneration.from_pretrained("facebook/rag-sequence-nq", use_dummy_dataset=True) >>> # 1. Encode >>> question_hidden_states = model.question_encoder(input_ids)[0] >>> # 2. Retrieve >>> docs_dict = retriever(input_ids.numpy(), question_hidden_states.detach().numpy(), return_tensors="pt") >>> doc_scores = torch.bmm( ... question_hidden_states.unsqueeze(1), docs_dict["retrieved_doc_embeds"].float().transpose(1, 2) ... ).squeeze(1) >>> # 3. Forward to generator >>> outputs = model( ... context_input_ids=docs_dict["context_input_ids"], ... context_attention_mask=docs_dict["context_attention_mask"], ... doc_scores=doc_scores, ... decoder_input_ids=labels, ... ) ```NFrmrnrorprqrrrrrrrsrtrurv)repsilonrrvrrrrrrrrr r!r"r#r$r%r&r'r()rOrvrrr5get_nllrrlabel_smoothingrrrrrrr r!r"r#r$r%r&r'r()rjrmrnrorprqrrrrrrrsrtrurrrrvr:outputsrr1r1r2rsjG z RagSequenceForGeneration.forwardcCs|jjSrrrjr1r1r2r?}sz"RagSequenceForGeneration.retrievercCs|jjSrr5rTrr1r1r2rTsz"RagSequenceForGeneration.generatorcCs|jjSrr5rSrr1r1r2rSsz)RagSequenceForGeneration.question_encoder) rmrnrrrdo_deduplicationnum_return_sequences num_beamsrvr@c Ks\| dk r | n|jj} |dk r |n|jj}|dk r4|n|jj} |dk rH|n|jj}|dk sh|dk shtd|jdk r|dkr|j||dd} |j||  t j  |jjj| ddd}| |}g} || d<|| d <d| d <|dk r|jdn |jd| }t|D]8}||| |d | }|jj|f| }|r\t td d |D}|jd}|dk r|||d |d }|||dd}n|dk std|dk std||d }||| |d | }||d }|||d ddf}||d }|||||dd}|d | d }| ||q |j| |jjjdS)a Implements RAG sequence "thorough" decoding. Read the [`~generation.GenerationMixin.generate`]` documentation for more information on how to set other generate input parameters. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): The sequence used as a prompt for the generation. If `input_ids` is not passed, then `context_input_ids` has to be provided. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) context_input_ids (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Input IDs post-processed from the retrieved documents and the question encoder input_ids by the retriever. context_attention_mask (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Attention mask post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model is not initialized with a `retriever` or `input_ids` is not given, `context_input_ids` and `context_attention_mask` have to be provided to the forward pass. They are returned by [`~RagRetriever.__call__`]. doc_scores (`torch.FloatTensor` of shape `(batch_size, config.n_docs)`): Score between each retrieved document embeddings (see `retrieved_doc_embeds`) and `question_encoder_last_hidden_state`. If the model is not initialized with a `retriever` or `input_ids` is not given, `doc_scores` has to be provided to the forward pass. `doc_scores` are returned by [`~RagRetriever.__call__`]. do_deduplication (`bool`, *optional*): Whether or not to deduplicate the generations from different context documents for a given input. Has to be set to `False` if used while training with distributed backend. num_return_sequences(`int`, *optional*, defaults to 1): The number of independently computed returned sequences for each element in the batch. Note that this is not the value we pass to the `generator`'s `[`~generation.GenerationMixin.generate`]` function, where we set `num_return_sequences` to `num_beams`. num_beams (`int`, *optional*, defaults to 1): Number of beams for beam search. 1 means no beam search. n_docs (`int`, *optional*, defaults to `config.n_docs`) Number of documents to retrieve and/or number of documents for which to generate an answer. kwargs (`Dict[str, Any]`, *optional*): Additional kwargs will be passed to [`~generation.GenerationMixin.generate`]. Return: `torch.LongTensor` of shape `(batch_size * num_return_sequences, sequence_length)`: The generated sequences. The second dimension (sequence length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. Nz= At least one of input_ids or context_input_ids must be givenrnrrxryrrrrnrcSsi|]}t||qSr1)r`tolist)rFkr1r1r2rIs z5RagSequenceForGeneration.generate..T)rrzMake sure that `context_attention_mask` are passed, if no `input_ids` is set. Alternatively, you can set a retriever using the `set_retriever(...)` function.zMake sure that `doc_scores` are passed, if no `input_ids` is set. Alternatively, you can set a retriever using the `set_retriever(...)` function.)rrrrrr) pad_token_id)rOrvrrrrXr?rSrrrr-rrrTrzrrangegeneratestacklistvaluesrepeatZtopkappend _cat_and_padr)rjrmrnrrrrrrrv model_kwargsZnum_doc_return_sequencesquestion_hidden_statesZhypos batch_sizeindexZgenerator_input_idsZoutput_sequencesZnum_candidatesZ new_input_idsrZindividual_input_idsZindividual_attention_maskZindividual_doc_scoresZ top_cand_indsr1r1r2rsA       z!RagSequenceForGeneration.generateFcsHtddddfjddjjjgd|dk rF|njj}jj p^jjj }|dk odddf | } fdd} t j j|dd|jd||d|d} t j j|dddd} | ddddddddf} | ddddddddf}| ddddddddf}tj| || |gdd}ddd|dd|kst|jdd}|jdd d }| ||\}}|r| r|ddddddfdn|d}|d}|d}|d}| }| }|r"|}|}||d}d ||||}|S) NrrcsDjjj}|r0||d||d|d|dfSNrr{eqrOrTranyZ masked_fill_rll smooth_objZpad_maskrjtargetr1r2 _mask_pads.s   z4RagSequenceForGeneration.get_nll.._mask_padsr{rrLrrTrZkeepdim?)r-catnewrfill_rOrTrrv bos_token_idrallr functional log_softmaxrsizerrrrXgathersum logsumexp)rj seq_logitsrrrrrrvrZuse_bosr seq_logprobs doc_logprobsZfirst_token_scoresZsecond_token_scores remainder rag_logprobsrrnll_loss smooth_losseps_irr1rr2r sH0"    6   z RagSequenceForGeneration.get_nllcCsv|dtdd|Dtdd|D|}d}|D]6}|||||jdd|jdf<||jd7}q:|S)NrcSsg|]}|jdqS)rrrFtr1r1r2 ^sz9RagSequenceForGeneration._cat_and_pad..cSsg|]}|jdqS)rrrr1r1r2r^sr)rrmaxrr)Ztensorsroutputindrr1r1r2r[s0$z%RagSequenceForGeneration._cat_and_pad)NNNN)NNNNNNNNNNNNNNNNN) NNNNNNNNN)FrFN)"r)r*r+rr rrrerrrrrrrr-r0rrrr.rrrpropertyr?rTrSno_gradrr staticmethodrrar1r1r;r2rs      ;rzo A RAG-token model implementation. It performs RAG-token specific marginalization in the forward pass. cseZdZd*eeeeeeeedfdd ZedddZedd d Z d+d d Z e d dZ e ddZ e ddZeddZd,ddZeeeeedd-eejeejeeeejeejeejeeeejeejeejeejeeeeeeeeeeeeeejeeedddZe dddddddde!e"f eejeejeejeejeejeeee#e$eejge%efee!ee"ejd ddZ&ddZ'd d!Z(d"d#Z)d.d$d%Z*d/d(d)Z+Z,S)0RagTokenForGenerationNrcc s^|dk s |dk r|dk s td|dkrRagTokenForGeneration._reorder_cache.._reorder_stackedr1c3s |]}||jVqdSr)rdevice)rFZ past_staterbeam_idxr1r2 sz7RagTokenForGeneration._reorder_cache..)tuple)rrZreordered_pastZ layer_pastr1rr2_reorder_cachesz$RagTokenForGeneration._reorder_cachecCsp|dk r |n|jj}tjj|dd|jd||d|d}tj|dd}|| d d}tj |ddS)Nr{rrr) rOrvrrrrrrr-rr)rjrrrvrrZ log_prob_sumr1r1r2 marginalizes z!RagTokenForGeneration.marginalizerk)rmrnrorprqrrrrrrrsrtrurrrrvr@cKs|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|dk rT|dkrP|}d} |j|||||||| || | | | |d}d}|j}|dk r|dk st|j|j|j|||jj |d}|r| ||j|}t |||j|j |j |j|j|j|j|j|j|j|j|j|j|j|jdS)ay do_marginalize (`bool`, *optional*): If `True`, the logits are marginalized over all documents by making use of `torch.nn.functional.log_softmax`. reduce_loss (`bool`, *optional*): Only relevant if `labels` is passed. If `True`, the NLL loss is reduced using the `torch.Tensor.sum` operation. kwargs (`Dict[str, any]`, optional, defaults to *{}*): Legacy dictionary, which is required so that model can use *generate()* function. Returns: Example: ```python >>> from transformers import AutoTokenizer, RagRetriever, RagTokenForGeneration >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("facebook/rag-token-nq") >>> retriever = RagRetriever.from_pretrained( ... "facebook/rag-token-nq", index_name="exact", use_dummy_dataset=True ... ) >>> # initialize with RagRetriever to do everything in one forward call >>> model = RagTokenForGeneration.from_pretrained("facebook/rag-token-nq", retriever=retriever) >>> inputs = tokenizer("How many people live in Paris?", return_tensors="pt") >>> targets = tokenizer(text_target="In Paris, there are 10 million people.", return_tensors="pt") >>> input_ids = inputs["input_ids"] >>> labels = targets["input_ids"] >>> outputs = model(input_ids=input_ids, labels=labels) >>> # or use retriever separately >>> model = RagTokenForGeneration.from_pretrained("facebook/rag-token-nq", use_dummy_dataset=True) >>> # 1. Encode >>> question_hidden_states = model.question_encoder(input_ids)[0] >>> # 2. Retrieve >>> docs_dict = retriever(input_ids.numpy(), question_hidden_states.detach().numpy(), return_tensors="pt") >>> doc_scores = torch.bmm( ... question_hidden_states.unsqueeze(1), docs_dict["retrieved_doc_embeds"].float().transpose(1, 2) ... ).squeeze(1) >>> # 3. Forward to generator >>> outputs = model( ... context_input_ids=docs_dict["context_input_ids"], ... context_attention_mask=docs_dict["context_attention_mask"], ... doc_scores=doc_scores, ... decoder_input_ids=labels, ... ) >>> # or directly generate >>> generated = model.generate( ... context_input_ids=docs_dict["context_input_ids"], ... context_attention_mask=docs_dict["context_attention_mask"], ... doc_scores=doc_scores, ... ) >>> generated_string = tokenizer.batch_decode(generated, skip_special_tokens=True) ```NFr)rrrvr)rOrvrrr5rrXrrrrrrrrrrr r!r"r#r$r%r&r'r()rjrmrnrorprqrrrrrrrsrtrurrrrvr:rrrr1r1r2rspO  zRagTokenForGeneration.forward) rmrnrrrrvgeneration_configprefix_allowed_tokens_fnlogits_processorstopping_criteriar@c  s|dkr|j}t|}|jf| } dk r0n|jj|jdk r|dkr|j||dd} |j||  t j  |jjjdd}|d|d|d}}}| | }| |}| |}t | d |d d d }|jddkstd d |jdd |jd|jj}|||dd}t j|jd f|jt jt|jd}|jd}|d}d"fdd }|||jd}|||jd|d<|j|jdd}|| d<|| d<|| d<| d<|j ||||| d}|jd kr8|j!d krt"d|j!d|j#|f||j$|j%|j&d| S|jd kr|j!|jkrZt"dt'|j|j|j(|j)|j!|j$d }|j*||f||j$|j%|j&d| St"d!|jdS)#a Implements RAG token decoding. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): The sequence used as a prompt for the generation. If `input_ids` is not passed, then `context_input_ids` has to be provided. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) context_input_ids (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Input IDs post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model has is not initialized with a `retriever`, `context_input_ids` has to be provided to the forward pass. `context_input_ids` are returned by [`~RagRetriever.__call__`]. context_attention_mask (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Attention mask post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model has is not initialized with a `retriever`, `context_input_ids` has to be provided to the forward pass. `context_input_ids` are returned by [`~RagRetriever.__call__`]. doc_scores (`torch.FloatTensor` of shape `(batch_size, config.n_docs)`): Score between each retrieved document embeddings (see `retrieved_doc_embeds`) and `question_encoder_last_hidden_state`. If the model has is not initialized with a `retriever`, `context_input_ids` has to be provided to the forward pass. `context_input_ids` are returned by [`~RagRetriever.__call__`]. n_docs (`int`, *optional*, defaults to `config.n_docs`) Number of documents to retrieve and/or number of documents for which to generate an answer. generation_config (`~generation.GenerationConfig`, *optional*): The generation configuration to be used as base parametrization for the generation call. `**kwargs` passed to generate matching the attributes of `generation_config` will override them. If `generation_config` is not provided, the default will be used, which has the following loading priority: 1) from the `generation_config.json` model file, if it exists; 2) from the model configuration. Please note that unspecified parameters will inherit [`~generation.GenerationConfig`]'s default values, whose documentation should be checked to parameterize generation. prefix_allowed_tokens_fn (`Callable[[int, torch.Tensor], List[int]]`, *optional*): If provided, this function constraints the beam search to allowed tokens only at each step. If not provided no constraint is applied. This function takes 2 arguments `inputs_ids` and the batch ID `batch_id`. It has to return a list with the allowed tokens for the next generation step conditioned on the previously generated tokens `inputs_ids` and the batch ID `batch_id`. This argument is useful for constrained generation conditioned on the prefix, as described in [Autoregressive Entity Retrieval](https://arxiv.org/abs/2010.00904). logits_processor (`LogitsProcessorList`, *optional*): Custom logits processors that complement the default logits processors built from arguments and a model's config. If a logit processor is passed that is already created with the arguments or a model's config an error is thrown. stopping_criteria (`StoppingCriteriaList`, *optional*): Custom stopping criteria that complement the default stopping criteria built from arguments and a model's config. If a stopping criteria is passed that is already created with the arguments or a model's config an error is thrown. kwargs (`Dict[str, Any]`, *optional*): Ad hoc parametrization of `generate_config` and/or additional model-specific kwargs that will be forwarded to the `forward` function of the model. Return: `torch.LongTensor` of shape `(batch_size * num_return_sequences, sequence_length)`: The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. NrrrxryrrrrrLr|r}r~T)rmrnrw)Zdtyperr{last_hidden_statecsl|ddddfdf|jdd}||f|jdd}||f|jddS)Nrr )Zreshaperexpand)Ztensorrrrvr1r2extend_enc_outputs,z9RagTokenForGeneration.generate..extend_enc_output)rrrrornrv)rinput_ids_seq_lengthZencoder_input_idsrrz)num_return_sequences has to be 1, but is z when doing greedy search.)r max_lengthr eos_token_idzA`num_return_sequences` has to be smaller or equal to `num_beams`.)rrrlength_penaltyZdo_early_stoppingZnum_beam_hyps_to_keepruH`num_beams` has to be an integer strictly superior to 0 (≥ 1), but is )N)+rcopydeepcopyupdaterOrvr?rSrrrr-rrrTrzrrrrrrXr5Z get_encoderfullrdecoder_start_token_idlongnext parametersrrZ_get_logits_processorr ValueErrorZ greedy_searchrrrr rZearly_stoppingZ beam_search)rjrmrnrrrrvrrrrr:rroutrencoderrorrrZ pre_processorZ beam_scorerr1rr2r^sQ                   zRagTokenForGeneration.generatecCs |jjSr)r5rTget_input_embeddingsrr1r1r2r+sz*RagTokenForGeneration.get_input_embeddingscCs |jjSr)r5rTget_output_embeddingsrr1r1r2r.sz+RagTokenForGeneration.get_output_embeddingscCs|jj|Sr)r5rTset_output_embeddings)rjZnew_embeddingsr1r1r2r1sz+RagTokenForGeneration.set_output_embeddingscCsX|dkr|jj}||j}|ddddf|ddddf<||dddf<|S)zCShift input ids one token to the right, and pad with start_token_idNr{rr)rOr Z new_zerosrclone)rjrmZstart_token_idZshifted_input_idsr1r1r2shift_tokens_right4s  (z(RagTokenForGeneration.shift_tokens_rightFrcs |dk r |njj}tddddfjddjjjgdfdd} |||} d | kst |j dd} |jddd} || | \} } | d} | d} | } | } |r| } | } ||d} d || | | }|S) NrrcsDjjj}|r0||d||d|d|dfSrrrrr1r2rDs   z1RagTokenForGeneration.get_nll.._mask_padsr{rTrr)rOrvr-rrrrrTrrrrrXrrr)rjrrrrrrvrrrrrrrrr1rr2r=s,0   zRagTokenForGeneration.get_nll)NNNN)NNNNNN)N)NNNNNNNNNNNNNNNNN)N)FrN)-r)r*r+rr rrrerrrrr?rTrSrrrrrrrrr-r0r.rrrrrrrr rr rrrrrrrrrar1r1r;r2rgs         M r)+r,r dataclassesrtypingrrrrrr-rZconfiguration_utilsr Z generationr r r rZmodeling_outputsrZmodeling_utilsrutilsrrrZconfiguration_ragrZ retrieval_ragrZ get_loggerr)loggerrrr3r4ZRAG_START_DOCSTRINGrrbrrr1r1r1r2sL        ^Y&&>q