U ,-ep@sdZddlmZddlmZmZmZddlZddlmZm Z ddl m Z ddl m Z dd lmZmZmZmZmZd d lmZmZd d lmZeeZdZdZddgZddgZddgZ eGdddeZ!eGdddeZ"eGdddeZ#Gddde Z$Gddde$Z%Gdd d e$Z&Gd!d"d"e$Z'Gd#d$d$e$Z(Gd%d&d&e$Z)d'Z*d(Z+d)Z,ed*e*Gd+d,d,e'Z-ed-e*Gd.d/d/e(Z.ed0e*Gd1d2d2e)Z/dS)3z6 PyTorch DPR model for Open Domain Question Answering.) dataclass)OptionalTupleUnionN)Tensornn)BaseModelOutputWithPooling)PreTrainedModel) ModelOutputadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) BertEncoder BertModel) DPRConfigrz'facebook/dpr-ctx_encoder-single-nq-basez&facebook/dpr-ctx_encoder-multiset-basez,facebook/dpr-question_encoder-single-nq-basez+facebook/dpr-question_encoder-multiset-basez"facebook/dpr-reader-single-nq-basez!facebook/dpr-reader-multiset-basec@sHeZdZUdZejed<dZee ejed<dZ ee ejed<dS)DPRContextEncoderOutputac Class for outputs of [`DPRQuestionEncoder`]. Args: pooler_output (`torch.FloatTensor` of shape `(batch_size, embeddings_size)`): The DPR encoder outputs the *pooler_output* that corresponds to the context representation. Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer. This output is to be used to embed contexts for nearest neighbors queries with questions embeddings. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. pooler_outputN hidden_states attentions __name__ __module__ __qualname____doc__torch FloatTensor__annotations__rrrrr!r!e/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/models/dpr/modeling_dpr.pyr=s  rc@sHeZdZUdZejed<dZee ejed<dZ ee ejed<dS)DPRQuestionEncoderOutputac Class for outputs of [`DPRQuestionEncoder`]. Args: pooler_output (`torch.FloatTensor` of shape `(batch_size, embeddings_size)`): The DPR encoder outputs the *pooler_output* that corresponds to the question representation. Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer. This output is to be used to embed questions for nearest neighbors queries with context embeddings. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. rNrrrr!r!r!r"r#Ys  r#c@sdeZdZUdZejed<dZejed<dZejed<dZ e e ejed<dZ e e ejed<dS)DPRReaderOutputa Class for outputs of [`DPRQuestionEncoder`]. Args: start_logits (`torch.FloatTensor` of shape `(n_passages, sequence_length)`): Logits of the start index of the span for each passage. end_logits (`torch.FloatTensor` of shape `(n_passages, sequence_length)`): Logits of the end index of the span for each passage. relevance_logits (`torch.FloatTensor` of shape `(n_passages, )`): Outputs of the QA classifier of the DPRReader that corresponds to the scores of each passage to answer the question, compared to all the other passages. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. start_logitsN end_logitsrelevance_logitsrr) rrrrrrr r&r'rrrrr!r!r!r"r$us  r$c@seZdZddZdddZdS)DPRPreTrainedModelcCst|tjr:|jjjd|jjd|jdk r|jj nft|tj rz|jjjd|jjd|j dk r|jj|j  n&t|tj r|jj |jj ddS)zInitialize the weightsg)ZmeanZstdNg?) isinstancerLinearweightdataZnormal_configZinitializer_rangeZbiasZzero_Z EmbeddingZ padding_idxZ LayerNormZfill_)selfmoduler!r!r" _init_weightss      z DPRPreTrainedModel._init_weightsFcCst|tr||_dSN)r)rZgradient_checkpointing)r.r/valuer!r!r"_set_gradient_checkpointings z.DPRPreTrainedModel._set_gradient_checkpointingN)F)rrrr0r3r!r!r!r"r(sr(c sreZdZdZedfdd Zdeeeeeeeeeee e e edffdd d Z e ed d d ZZS) DPREncoder bert_modelr-csdt|t|dd|_|jjjdkr0td|j|_|jdkrXt |jjj|j|_ | dS)NF)Zadd_pooling_layerrz!Encoder hidden_size can't be zero) super__init__rr5r- hidden_size ValueErrorprojection_dimrr* encode_proj post_initr.r- __class__r!r"r8s  zDPREncoder.__init__NF. input_idsattention_masktoken_type_ids inputs_embedsoutput_attentionsoutput_hidden_states return_dictreturnc Csv|j|||||||d}|d} | dddddf} |jdkrJ|| } |sb| | f|ddSt| | |j|jdS)NrBrCrDrErFrGrHrr)Zlast_hidden_staterrr)r5r;r<r rr) r.rBrCrDrErFrGrHoutputssequence_outputZ pooled_outputr!r!r"forwards*    zDPREncoder.forward)rIcCs|jdkr|jjS|jjjS)Nr)r;r<Z out_featuresr5r-r9)r.r!r!r"embeddings_sizes zDPREncoder.embeddings_size)NNNFFF)rrrbase_model_prefixrr8rrboolrr rrMpropertyintrN __classcell__r!r!r?r"r4s( #r4c sVeZdZdZedfdd Zd eeeeeeee e e edffdd d Z Z S) DPRSpanPredictorencoderr6csFt|t||_t|jjd|_t|jjd|_| dS)Nrr) r7r8r4rUrr*rN qa_outputs qa_classifierr=r>r?r!r"r8s   zDPRSpanPredictor.__init__NF.rBrCrErFrGrHrIcCs|dk r|n|dd\}}|j||||||d} | d} || } | jddd\} } | d} | d} || dddddf}| ||} | ||} ||}|s| | |f| ddSt| | || j | j dS)Nr)rCrErFrGrHrr)dim)r%r&r'rr) sizerUrVsplitZsqueeze contiguousrWviewr$rr)r.rBrCrErFrGrHZ n_passagesZsequence_lengthrKrLZlogitsr%r&r'r!r!r"rMs6 $    zDPRSpanPredictor.forward)NFFF)rrrrOrr8rrrPrr$rrMrSr!r!r?r"rTs rTc@seZdZdZeZdZdZdS)DPRPretrainedContextEncoder An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. N ctx_encoderrrrrr config_classZload_tf_weightsrOr!r!r!r"r_"sr_c@seZdZdZeZdZdZdS)DPRPretrainedQuestionEncoderr`Nquestion_encoderrbr!r!r!r"rd-srdc@seZdZdZeZdZdZdS)DPRPretrainedReaderr`Nspan_predictorrbr!r!r!r"rf8srfa= This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`DPRConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a" Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. To match pretraining, DPR input sequence should be formatted with [CLS] and [SEP] tokens as follows: (a) For sequence pairs (for a pair title+text for example): ``` tokens: [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP] token_type_ids: 0 0 0 0 0 0 0 0 1 1 1 1 1 1 ``` (b) For single sequences (for a question for example): ``` tokens: [CLS] the dog is hairy . [SEP] token_type_ids: 0 0 0 0 0 0 0 ``` DPR is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a? Args: input_ids (`Tuple[torch.LongTensor]` of shapes `(n_passages, sequence_length)`): Indices of input sequence tokens in the vocabulary. It has to be a sequence triplet with 1) the question and 2) the passages titles and 3) the passages texts To match pretraining, DPR `input_ids` sequence should be formatted with [CLS] and [SEP] with the format: `[CLS] [SEP] [SEP] ` DPR is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than the left. Indices can be obtained using [`DPRReaderTokenizer`]. See this class documentation for more details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `(n_passages, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) inputs_embeds (`torch.FloatTensor` of shape `(n_passages, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z\The bare DPRContextEncoder transformer outputting pooler outputs as context representations.cseZdZedfdd Zeeeee dd e e e e e e e e e e e e e e e eee dffddd ZZS) DPRContextEncoderr6cs(t|||_t||_|dSr1)r7r8r-r4rar=r>r?r!r"r8s  zDPRContextEncoder.__init__ output_typercN.rAc Cs |dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|dk rV|dk rVtdn4|dk rh|}n"|dk r|dd}ntd|dk r|jn|j} |dkr|dkrtj|| dn ||jj k}|dkrtj |tj | d}|j |||||||d} |s | ddSt | j| j| jd S) a Return: Examples: ```python >>> from transformers import DPRContextEncoder, DPRContextEncoderTokenizer >>> tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base") >>> model = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base") >>> input_ids = tokenizer("Hello, is my dog cute ?", return_tensors="pt")["input_ids"] >>> embeddings = model(input_ids).pooler_output ```NDYou cannot specify both input_ids and inputs_embeds at the same timerY5You have to specify either input_ids or inputs_embedsdeviceZdtypernrJrrrr)r-rFrGuse_return_dictr:r[rnrones pad_token_idzeroslongrarrrr r.rBrCrDrErFrGrH input_shapernrKr!r!r"rMsF     zDPRContextEncoder.forward)NNNNNNN)rrrrr8r DPR_ENCODERS_INPUTS_DOCSTRINGrr_CONFIG_FOR_DOCrrrPrrrMrSr!r!r?r"rhs( rhz^The bare DPRQuestionEncoder transformer outputting pooler outputs as question representations.cseZdZedfdd Zeeeee dd e e e e e e e e e e e e e e e eee dffddd ZZS) DPRQuestionEncoderr6cs(t|||_t||_|dSr1)r7r8r-r4rer=r>r?r!r"r8 s  zDPRQuestionEncoder.__init__riN.rAc Cs,|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|dk rV|dk rVtdn@|dk rt||||}n"|dk r|dd}ntd|dk r|jn|j} |dkr|dkrtj || dn ||jj k}|dkrtj |tj | d}|j |||||||d} |s| ddSt| j| j| jd S) a Return: Examples: ```python >>> from transformers import DPRQuestionEncoder, DPRQuestionEncoderTokenizer >>> tokenizer = DPRQuestionEncoderTokenizer.from_pretrained("facebook/dpr-question_encoder-single-nq-base") >>> model = DPRQuestionEncoder.from_pretrained("facebook/dpr-question_encoder-single-nq-base") >>> input_ids = tokenizer("Hello, is my dog cute ?", return_tensors="pt")["input_ids"] >>> embeddings = model(input_ids).pooler_output ``` NrkrYrlrmrorJrrp)r-rFrGrqr:%warn_if_padding_and_no_attention_maskr[rnrrrrsrtrurer#rrrrvr!r!r"rMsH      zDPRQuestionEncoder.forward)NNNNNNN)rrrrr8r rxrr#ryrrrPrrrMrSr!r!r?r"rzs( rzz;The bare DPRReader transformer outputting span predictions.cszeZdZedfdd Zeeeee dd e e e e e e e e e e e e e eee dffddd ZZS) DPRReaderr6cs(t|||_t||_|dSr1)r7r8r-rTrgr=r>r?r!r"r8]s  zDPRReader.__init__riN.rXc Cs|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|dk rV|dk rVtdn@|dk rt||||}n"|dk r|dd}ntd|dk r|jn|j}|dkrtj ||d}|j ||||||dS)a Return: Examples: ```python >>> from transformers import DPRReader, DPRReaderTokenizer >>> tokenizer = DPRReaderTokenizer.from_pretrained("facebook/dpr-reader-single-nq-base") >>> model = DPRReader.from_pretrained("facebook/dpr-reader-single-nq-base") >>> encoded_inputs = tokenizer( ... questions=["What is love ?"], ... titles=["Haddaway"], ... texts=["'What Is Love' is a song recorded by the artist Haddaway"], ... return_tensors="pt", ... ) >>> outputs = model(**encoded_inputs) >>> start_logits = outputs.start_logits >>> end_logits = outputs.end_logits >>> relevance_logits = outputs.relevance_logits ``` NrkrYrlrm)rErFrGrH) r-rFrGrqr:r{r[rnrrrrg) r.rBrCrErFrGrHrwrnr!r!r"rMds.!   zDPRReader.forward)NNNNNN)rrrrr8r DPR_READER_INPUTS_DOCSTRINGrr$ryrrrPrrrMrSr!r!r?r"r|Xs$ r|)0r dataclassesrtypingrrrrrrZmodeling_outputsr Zmodeling_utilsr utilsr r r rrZbert.modeling_bertrrZconfiguration_dprrZ get_loggerrloggerryZ_CHECKPOINT_FOR_DOCZ1DPR_CONTEXT_ENCODER_PRETRAINED_MODEL_ARCHIVE_LISTZ2DPR_QUESTION_ENCODER_PRETRAINED_MODEL_ARCHIVE_LISTZ(DPR_READER_PRETRAINED_MODEL_ARCHIVE_LISTrr#r$r(r4rTr_rdrfZDPR_START_DOCSTRINGrxr}rhrzr|r!r!r!r"sf       8>  8%MN