U 9%e^o@sdZddlZddlZddlZddlmZmZmZmZddl Z ddl m Z ddl mZddlmZmZmZmZmZdd lmZdd lmZerdd lmZmZmZerddlZeeZ d Z!Gd ddZ"Gddde"Z#Gddde"Z$Gddde$Z%Gddde$Z&GdddZ'dS)z#RAG Retriever model implementation.N)IterableListOptionalTuple)PreTrainedTokenizer) BatchEncoding) cached_fileis_datasets_availableis_faiss_availableloggingrequires_backends) RagConfig) RagTokenizer)Dataset load_datasetload_from_diskzAhttps://storage.googleapis.com/huggingface-nlp/datasets/wiki_dpr/c@sVeZdZdZejeedddZdeje ejejfdddZ d d Z d d Z d S)IndexzL A base class for the Indices encapsulated by the [`RagRetriever`]. doc_idsreturncCstdS)z Returns a list of dictionaries, containing titles and text of the retrieved documents. Args: doc_ids (`np.ndarray` of shape `(batch_size, n_docs)`): A tensor of document indices. NNotImplementedErrorselfrrd/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/transformers/models/rag/retrieval_rag.py get_doc_dicts1szIndex.get_doc_dictsquestion_hidden_statesrcCstdS)a$ For each query in the batch, retrieves `n_docs` documents. Args: question_hidden_states (`np.ndarray` of shape `(batch_size, vector_size)`): An array of query vectors. n_docs (`int`): The number of docs retrieved per query. Returns: `np.ndarray` of shape `(batch_size, n_docs)`: A tensor of indices of retrieved documents. `np.ndarray` of shape `(batch_size, vector_size)`: A tensor of vector representations of retrieved documents. Nr)rr!n_docsrrr get_top_docs;szIndex.get_top_docscCstdS)zA Returns `True` if index is already initialized. Nrrrrris_initializedKszIndex.is_initializedcCstdS)a  A function responsible for loading the index into memory. Should be called only once per training run of a RAG model. E.g. if the model is trained on multiple GPUs in a distributed setup, only one of the workers will load the index. Nrr$rrr init_indexQszIndex.init_indexN)r) __name__ __module__ __qualname____doc__npndarrayrdictrrr#r%r&rrrrr,s  rc@sxeZdZdZdZdZddZddZdd Zd d Z d d Z ddZ e j dddZde jee je jfdddZdS) LegacyIndexa An index which can be deserialized from the files built using https://github.com/facebookresearch/DPR. We use default faiss index parameters as specified in that repository. Args: vector_size (`int`): The dimension of indexed vectors. index_path (`str`): A path to a *directory* containing index files compatible with [`~models.rag.retrieval_rag.LegacyIndex`] z,hf_bert_base.hnswSQ8_correct_phi_128.c_indexzpsgs_w100.tsv.pklcCs,g|_||_||_||_d|_d|_dS)NF)index_id_to_db_id index_path_load_passagespassages vector_sizeindex_index_initialized)rr3r0rrr__init__is  zLegacyIndex.__init__cCstj|}zt||}Wn@tk rZd|d|d|d|d|d }t|YnX|rrtd|ntd|d||S) Nz Can't load 'z'. Make sure that: - 'zB' is a correct remote path to a directory containing a file named z - or 'z=' is the correct path to a directory containing a file named z. z loading file z from cache at )ospathisdirr EnvironmentErrorloggerinfo)rr0filenameis_localZresolved_archive_filemsgrrr _resolve_pathqs "zLegacyIndex._resolve_pathc CsFtd|j||j|j}t|d}t|}W5QRX|S)NLoading passages from rb)r;r<r0r@PASSAGE_FILENAMEopenpickleload)r passages_pathZ passages_filer2rrrr1s  zLegacyIndex._load_passagesc Cstd|j||j|jd}t||_||j|jd}t|d}t ||_ W5QRXt |j |jj kstddS)NLoading index from z .index.dprz.index_meta.dprrBzsz-LegacyIndex.get_doc_dicts..csg|]}j|qSr)r2rUr$rrrWscSsg|] }|dqS)rrrVdocrrrrWstitlecSsg|] }|dqSrrrXrrrrWstext)append)rrZdoc_listZ doc_ids_iidsdocsZ doc_dictsZdoc_dictrr$rrs  zLegacyIndex.get_doc_dictsrr c srtjt|dddd}t||f}j||\}}fdd|D}fdd|D}t|t|fS)Nfloat32)dtypercsg|]}fdd|DqS)cs$g|]}jt|ddqS)Nrb)r4Z reconstructrTrUr$rrrWs7LegacyIndex.get_top_docs...rrVrr$rrrWsz,LegacyIndex.get_top_docs..csg|]}fdd|DqS)csg|]}tj|qSr)rTr/rUr$rrrWsrcrrdr$rrrWs)r+zerosrKZreshapeZhstackr4searcharray) rr!r"Zaux_dimZquery_nhsw_vectors_Zdocs_idsvectorsr^rr$rr#s zLegacyIndex.get_top_docsN)r)r'r(r)r*rIrCr6r@r1rNr%r&r+rgrr,rr#rrrrr.Zs  r.c@sjeZdZdddZedddZddZd d Zej e e d d d Z dej e ej ej fdddZdS) HFIndexBaseFcCs6||_||_||_|j|d|jddgddddS)N with_indexnumpy embeddingsTr`)columnsoutput_all_columnsra)r3datasetr5_check_dataset_format set_format)rr3rqindex_initializedrrrr6s  zHFIndexBase.__init__rkcCsnt|jts tdt|jtdddht|jjdkrPtd|jj|rjd|jkrjtddS)Nz5Dataset should be a datasets.Dataset object, but got rZr\rnrzDataset should be a dataset with the following columns: title (str), text (str) and embeddings (arrays of dimension vector_size), but got columns zMissing faiss index in the dataset. Make sure you called `dataset.add_faiss_index` to compute it or `dataset.load_faiss_index` to load one from the disk.) isinstancerqr ValueErrortyperKsetZ column_namesZ list_indexes)rrlrrrrrs  z!HFIndexBase._check_dataset_formatcCs tdSrOrr$rrrr&szHFIndexBase.init_indexcCs|jSrOrPr$rrrr%szHFIndexBase.is_initializedrcsfddtjdDS)Ncsg|]}j|qSr)rqtolistrVirrrrrWsz-HFIndexBase.get_doc_dicts..r)rangeshaperrr|rrszHFIndexBase.get_doc_dictsrr c sjd||\}}fdd|D}dd|D}tt|D]B}t|||kr@t||t|t||jfg||<q@t|t|fS)Nrncs g|]}jdd|DqS)cSsg|]}|dkr|qSr[rrzrrrrWsz7HFIndexBase.get_top_docs...)rq)rVindicesr$rrrWsz,HFIndexBase.get_top_docs..cSsg|] }|dqS)rnrrXrrrrWs) rqZ search_batchr}rKr+Zvstackrer3rg)rr!r"rhr^r_rir{rr$rr#s0zHFIndexBase.get_top_docsN)F)r)r'r(r)r6boolrrr&r%r+r,rr-rrr#rrrrrjs  rjcs@eZdZdZd eeeeeeedfdd Zd d ZZ S) CanonicalHFIndexa A wrapper around an instance of [`~datasets.Datasets`]. If `index_path` is set to `None`, we load the pre-computed index available with the [`~datasets.arrow_dataset.Dataset`], otherwise, we load the index from the indicated path on disk. Args: vector_size (`int`): the dimension of the passages embeddings used by the index dataset_name (`str`, optional, defaults to `wiki_dpr`): A dataset identifier of the indexed dataset on HuggingFace AWS bucket (list all available datasets and ids with `datasets.list_datasets()`). dataset_split (`str`, optional, defaults to `train`) Which split of the `dataset` to load. index_name (`str`, optional, defaults to `train`) The index_name of the index associated with the `dataset`. The index loaded from `index_path` will be saved under this name. index_path (`str`, optional, defaults to `None`) The path to the serialized faiss index on disk. use_dummy_dataset (`bool`, optional, defaults to `False`): If True, use the dummy configuration of the dataset for tests. wiki_dprtrainNF)r3 dataset_name dataset_split index_namer0cst|dkt|dkdkr$td||_||_||_||_||_td|jt |jd|j|jd}t j ||dddS)Nrz,Please provide `index_name` or `index_path`.rAF)rlsplitdummyrt) rTrvrrrr0use_dummy_datasetr;r<rsuperr6)rr3rrrr0rrq __class__rrr6s zCanonicalHFIndex.__init__cCs|jdk r0td|j|jjd|jdnLtd|jd|jt|jdd|j|j|j d|_|jj ddgddd|_ dS) NrHrnfilez with index name T)Zwith_embeddingsrlrrrrm)rorp) r0r;r<rqload_faiss_indexrrrrrrsr5r$rrrr& s zCanonicalHFIndex.init_index)rrNNF) r'r(r)r*rTrSrr6r& __classcell__rrrrrsrcs<eZdZdZd edfdd ZeddZdd ZZ S) CustomHFIndexa A wrapper around an instance of [`~datasets.Datasets`]. The dataset and the index are both loaded from the indicated paths on disk. Args: vector_size (`int`): the dimension of the passages embeddings used by the index dataset_path (`str`): The path to the serialized dataset on disk. The dataset should have 3 columns: title (str), text (str) and embeddings (arrays of dimension vector_size) index_path (`str`) The path to the serialized faiss index on disk. N)r3cs tj|||dkd||_dS)Nr)rr6r0)rr3rqr0rrrr6-szCustomHFIndex.__init__cCs>td||dks |dkr(tdt|}||||dS)NrAzPlease provide `dataset_path` and `index_path` after calling `dataset.save_to_disk(dataset_path)` and `dataset.get_index('embeddings').save(index_path)`.)r3rqr0)r;r<rvr)clsr3 dataset_pathr0rqrrrr1szCustomHFIndex.load_from_diskcCs6|s2td|j|jjd|jdd|_dS)NrHrnrT)r%r;r<r0rqrr5r$rrrr&<szCustomHFIndex.init_index)N) r'r(r)r*rTr6 classmethodrr&rrrrrrs   rcseZdZdZdfdd ZeddZeddd Zd d Z d d Z d ddZ e e ee dddZeje eejejfdddZeje eejeefdddZedddZd!eee ejedddZZS)" RagRetrievera Retriever used to get documents from vector queries. It retrieves the documents embeddings as well as the documents contents, and it formats them to be used with a RagModel. Args: config ([`RagConfig`]): The configuration of the RAG model this Retriever is used with. Contains parameters indicating which `Index` to build. You can load your own custom dataset with `config.index_name="custom"` or use a canonical one (default) from the datasets library with `config.index_name="wiki_dpr"` for example. question_encoder_tokenizer ([`PreTrainedTokenizer`]): The tokenizer that was used to tokenize the question. It is used to decode the question and then use the generator_tokenizer. generator_tokenizer ([`PreTrainedTokenizer`]): The tokenizer used for the generator part of the RagModel. index ([`~models.rag.retrieval_rag.Index`], optional, defaults to the one defined by the configuration): If specified, use this index instead of the one built using the configuration Examples: ```python >>> # To load the default "wiki_dpr" dataset with 21M passages from wikipedia (index name is 'compressed' or 'exact') >>> from transformers import RagRetriever >>> retriever = RagRetriever.from_pretrained( ... "facebook/dpr-ctx_encoder-single-nq-base", dataset="wiki_dpr", index_name="compressed" ... ) >>> # To load your own indexed dataset built with the datasets library. More info on how to build the indexed dataset in examples/rag/use_own_knowledge_dataset.py >>> from transformers import RagRetriever >>> dataset = ( ... ... ... ) # dataset must be a datasets.Datasets object with columns "title", "text" and "embeddings", and it must have a faiss index >>> retriever = RagRetriever.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base", indexed_dataset=dataset) >>> # To load your own indexed dataset built with the datasets library that was saved on disk. More info in examples/rag/use_own_knowledge_dataset.py >>> from transformers import RagRetriever >>> dataset_path = "path/to/my/dataset" # dataset saved via *dataset.save_to_disk(...)* >>> index_path = "path/to/my/index.faiss" # faiss index saved via *dataset.get_index("embeddings").save(...)* >>> retriever = RagRetriever.from_pretrained( ... "facebook/dpr-ctx_encoder-single-nq-base", ... index_name="custom", ... passages_path=dataset_path, ... index_path=index_path, ... ) >>> # To load the legacy index built originally for Rag's paper >>> from transformers import RagRetriever >>> retriever = RagRetriever.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base", index_name="legacy") ```NTcsn||_t|ddgt|p*|||_||_||_|j|_|j |_ ||_ |jr^| d|_ d|_dS)NdatasetsrJF)Z_init_retrievalr rr6 _build_indexr4generator_tokenizerquestion_encoder_tokenizerr"Zretrieval_batch_size batch_sizeconfiginit_retrievalctx_encoder_tokenizerreturn_tokenized_docs)rrrrr4rrrrr6ys zRagRetriever.__init__cCs`|jdkrt|j|jptS|jdkr.cat_input_and_doccsDg|]<}tD].}|d||d||qqS)rZr\)r})rVr{j)rr_ input_stringsr"rrrrWs z1RagRetriever.postprocess_docs.. max_lengthT)rreturn_tensorspadding truncation input_idsattention_mask)r}rKrZbatch_encode_plusrZmax_combined_length)rr_rrr"rZrag_input_stringsZcontextualized_inputsr)rr_rr"rrrpostprocess_docss   zRagRetriever.postprocess_docs)t chunk_sizercs fddtdtDS)Ncsg|]}||qSrrrzrrrrrWsz.RagRetriever._chunk_tensor..r)r}rK)rrrrrr _chunk_tensorszRagRetriever._chunk_tensor)r!r"rc Cs|||j}g}g}|D]R}t}|j||\}}tdt|d|j||||qt |t |fS)Nzindex search time: z sec, batch size ) rrtimer4r#r;debugr~extendr+rg) rr!r"Zquestion_hidden_states_batchedZ ids_batchedZvectors_batched start_timer^rirrr_main_retrieves  zRagRetriever._main_retrievecCs"|||\}}|||j|fS)a% Retrieves documents for specified `question_hidden_states`. Args: question_hidden_states (`np.ndarray` of shape `(batch_size, vector_size)`): A batch of query vectors to retrieve with. n_docs (`int`): The number of docs retrieved per query. Return: `Tuple[np.ndarray, np.ndarray, List[dict]]`: A tuple with the following objects: - **retrieved_doc_embeds** (`np.ndarray` of shape `(batch_size, n_docs, dim)`) -- The retrieval embeddings of the retrieved docs per query. - **doc_ids** (`np.ndarray` of shape `(batch_size, n_docs)`) -- The ids of the documents in the index - **doc_dicts** (`List[dict]`): The `retrieved_doc_embeds` examples per query. )rr4r)rr!r"rretrieved_doc_embedsrrrretrieveszRagRetriever.retrieve)rcCs||_d|_dS)NT)rr)rrrrrset_ctx_encoder_tokenizer/sz&RagRetriever.set_ctx_encoder_tokenizer)question_input_idsr!rcCs |dk r |n|j}|dk r|n|jjj}|||\}}}|jj|dd} |j|| |||d\} } |jrg} g} t t |D]>}t |D]0}| ||d|| ||d|qq||j | | dd|d}t | | |||d |d d |d St | | ||d |d SdS)a# Retrieves documents for specified `question_hidden_states`. Args: question_input_ids (`List[List[int]]`) batch of input ids question_hidden_states (`np.ndarray` of shape `(batch_size, vector_size)`: A batch of query vectors to retrieve with. prefix (`str`, *optional*): The prefix used by the generator's tokenizer. n_docs (`int`, *optional*): The number of docs retrieved per query. return_tensors (`str` or [`~utils.TensorType`], *optional*, defaults to "pt"): If set, will return tensors instead of list of python integers. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return Numpy `np.ndarray` objects. Returns: [`BatchEncoding`]: A [`BatchEncoding`] with the following fields: - **context_input_ids** -- List of token ids to be fed to a model. [What are input IDs?](../glossary#input-ids) - **context_attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names`). [What are attention masks?](../glossary#attention-mask) - **retrieved_doc_embeds** -- List of embeddings of the retrieved documents - **doc_ids** -- List of ids of the retrieved documents NT)Zskip_special_tokens)rr\rZlongest)rrrrr)context_input_idscontext_attention_maskrrZtokenized_doc_idsZtokenized_doc_attention_mask)Z tensor_type)rrrr)r"rrrrrZ batch_decoderrr}rKr]rr)rrr!rr"rrrr_rrrZretrieved_doc_textZretrieved_doc_titleZb_idxZdoc_idxZtokenized_docsrrr__call__4sV)   zRagRetriever.__call__)NT)N)N)NNN)r'r(r)r*r6 staticmethodrrrrrrrrTrrr+r,rrr-rrrrrrrrrrrCs*5   4 "  r)(r*r7rErtypingrrrrrmr+Ztokenization_utilsrZtokenization_utils_baserutilsr r r r r Zconfiguration_ragrZtokenization_ragrrrrrrJZ get_loggerr'r;rrr.rjrrrrrrrs,     .]*>$