U ,-e@sddZddlZddlZddlmZmZddlmZddlm Z ddl m Z e e ZGd d d ZdS) zTokenization classes for RAG.N)ListOptional) BatchEncoding)logging) RagConfigc @seZdZddZddZeddZddZd d Zd d Z d dZ ddZ de e ee e eeeee e eedddZdS) RagTokenizercCs||_||_|j|_dSN)question_encoder generatorcurrent_tokenizer)selfr r ri/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/models/rag/tokenization_rag.py__init__szRagTokenizer.__init__cCsbtj|rtd|dtj|ddtj|d}tj|d}|j||j|dS)NzProvided path (z#) should be a directory, not a fileT)exist_okquestion_encoder_tokenizergenerator_tokenizer) ospathisfile ValueErrormakedirsjoinr save_pretrainedr )rZsave_directoryZquestion_encoder_pathZgenerator_pathrrrr"s  zRagTokenizer.save_pretrainedcKsZddlm}|dd}|dkr*t|}|j||jdd}|j||jdd}|||dS)N) AutoTokenizerconfigr)rZ subfolderr)r r )Zauto.tokenization_autorpoprfrom_pretrainedr r )clsZpretrained_model_name_or_pathkwargsrrr r rrrr +s   zRagTokenizer.from_pretrainedcOs |j||Sr )r rargsr"rrr__call__=szRagTokenizer.__call__cOs|jj||Sr )r batch_decoder#rrrr&@szRagTokenizer.batch_decodecOs|jj||Sr )r decoder#rrrr'CszRagTokenizer.decodecCs |j|_dSr )r r rrrr_switch_to_input_modeFsz"RagTokenizer._switch_to_input_modecCs |j|_dSr )r r r(rrr_switch_to_target_modeIsz#RagTokenizer._switch_to_target_modeNlongestT) src_texts tgt_texts max_lengthmax_target_lengthpaddingreturn_tensors truncationreturnc Kstdt|dkr|jj}||fd||||d|} |dkrD| S|dkrT|jj}|f|d||||d|} | d| d<| S)Nu4`prepare_seq2seq_batch` is deprecated and will be removed in version 5 of 🤗 Transformers. Use the regular `__call__` method to prepare your inputs and the tokenizer under the `with_target_tokenizer` context manager to prepare your targets. See the documentation of your specific tokenizer for more detailsT)add_special_tokensr1r.r0r2)Z text_targetr4r1r0r.r2Z input_idslabels)warningswarn FutureWarningr Zmodel_max_length) rr,r-r.r/r0r1r2r"Z model_inputsr5rrrprepare_seq2seq_batchLsB   z"RagTokenizer.prepare_seq2seq_batch)NNNr+NT)__name__ __module__ __qualname__rr classmethodr r%r&r'r)r*rstrrintboolrr9rrrrr s2   r )__doc__rr6typingrrZtokenization_utils_baserutilsrZconfiguration_ragrZ get_loggerr:loggerr rrrrs