U ,È-eTã@s`dZddlmZmZmZddlmZddlmZm Z m Z m Z m Z ddl mZGdd„deƒZd S) z Processor class for Bros. é)ÚListÚOptionalÚUnioné)ÚProcessorMixin)Ú BatchEncodingÚPaddingStrategyÚPreTokenizedInputÚ TextInputÚTruncationStrategy)Ú TensorTypecsºeZdZdZdgZdZd‡fdd„ Zdeee e ee e fe ee e e fee e efeeeeeee ee e e e e e eee efed œd d „Zd d„Zdd„Zedd„ƒZ‡ZS)Ú BrosProcessora… Constructs a Bros processor which wraps a BERT tokenizer. [`BrosProcessor`] offers all the functionalities of [`BertTokenizerFast`]. See the docstring of [`~BrosProcessor.__call__`] and [`~BrosProcessor.decode`] for more information. Args: tokenizer (`BertTokenizerFast`): An instance of ['BertTokenizerFast`]. The tokenizer is a required input. Ú tokenizer)Z BertTokenizerZBertTokenizerFastNc s |dkrtdƒ‚tƒ |¡dS)Nz"You need to specify a `tokenizer`.)Ú ValueErrorÚsuperÚ__init__)ÚselfrÚkwargs©Ú __class__©úi/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/models/bros/processing_bros.pyr(szBrosProcessor.__init__TFr)ÚtextÚadd_special_tokensÚpaddingÚ truncationÚ max_lengthÚstrideÚpad_to_multiple_ofÚreturn_token_type_idsÚreturn_attention_maskÚreturn_overflowing_tokensÚreturn_special_tokens_maskÚreturn_offsets_mappingÚ return_lengthÚverboseÚreturn_tensorsÚreturncKs4|jf||||||||| | | | | ||dœ|—Ž}|S)z¶ This method uses [`BertTokenizerFast.__call__`] to prepare text for the model. Please refer to the docstring of the above two methods for more information. )rrrrrrrrr r!r"r#r$r%r&)r)rrrrrrrrrr r!r"r#r$r%r&rÚencodingrrrÚ__call__.s(ñðzBrosProcessor.__call__cOs|jj||ŽS)zÁ This method forwards all its arguments to BertTokenizerFast's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information. )rÚ batch_decode©rÚargsrrrrr*[szBrosProcessor.batch_decodecOs|jj||ŽS)z» This method forwards all its arguments to BertTokenizerFast's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. )rÚdecoder+rrrr-bszBrosProcessor.decodecCs|jj}tt |¡ƒS)N)rÚmodel_input_namesÚlistÚdictÚfromkeys)rZtokenizer_input_namesrrrr.iszBrosProcessor.model_input_names)N)NTFNNrNNNFFFFTN)Ú__name__Ú __module__Ú __qualname__Ú__doc__Ú attributesZtokenizer_classrrr r rÚboolÚstrrr rÚintr rr)r*r-Úpropertyr.Ú __classcell__rrrrr sR ð  î -r N)r5ÚtypingrrrZprocessing_utilsrZtokenization_utils_baserrr r r Úutilsr r rrrrÚs