U ,-e@s@dZddlZddlZddlmZddlmZGdddeZdS)z Processor class for Donut. N)contextmanager)ProcessorMixincsxeZdZdZddgZdZdZdfdd Zd d Zd d Z d dZ e ddZ dddZ eddZeddZZS)DonutProcessora Constructs a Donut processor which wraps a Donut image processor and an XLMRoBERTa tokenizer into a single processor. [`DonutProcessor`] offers all the functionalities of [`DonutImageProcessor`] and [`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]. See the [`~DonutProcessor.__call__`] and [`~DonutProcessor.decode`] for more information. Args: image_processor ([`DonutImageProcessor`]): An instance of [`DonutImageProcessor`]. The image processor is a required input. tokenizer ([`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]): An instance of [`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]. The tokenizer is a required input. image_processor tokenizerZAutoImageProcessorZ AutoTokenizerNc srd}d|kr"tdt|d}|dk r.|n|}|dkrBtd|dkrRtdt|||j|_d|_ dS)Nfeature_extractorzhThe `feature_extractor` argument is deprecated and will be removed in v5, use `image_processor` instead.z)You need to specify an `image_processor`.z"You need to specify a `tokenizer`.F) warningswarn FutureWarningpop ValueErrorsuper__init__rcurrent_processor_in_target_context_manager)selfrrkwargsr __class__k/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/models/donut/processing_donut.pyr,s zDonutProcessor.__init__cOs|jr|j||S|dd}|dd}t|dkrJ|d}|dd}|dkrb|dkrbtd|dk r||j|f||}|dk r|j|f|}|dkr|S|dkr|S|d|d<|SdS) a When used in normal mode, this method forwards all its arguments to AutoImageProcessor's [`~AutoImageProcessor.__call__`] and returns its output. If used in the context [`~DonutProcessor.as_target_processor`] this method forwards all its arguments to DonutTokenizer's [`~DonutTokenizer.__call__`]. Please refer to the doctsring of the above two methods for more information. imagesNtextrzBYou need to specify either an `images` or `text` input to process.Z input_idslabels)rrr lenr rr)rargsrrrinputs encodingsrrr__call__@s&      zDonutProcessor.__call__cOs|jj||S)z This method forwards all its arguments to DonutTokenizer's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information. )r batch_decoderrrrrrr!aszDonutProcessor.batch_decodecOs|jj||S)z This method forwards all its arguments to DonutTokenizer's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. )rdecoder"rrrr#hszDonutProcessor.decodeccs0tdd|_|j|_dV|j|_d|_dS)z Temporarily sets the tokenizer for processing the input. Useful for encoding the labels when fine-tuning TrOCR. z`as_target_processor` is deprecated and will be removed in v5 of Transformers. You can process your labels by using the argument `text` of the regular `__call__` method (either in the same call as your images inputs, or in a separate call.TNF)r r rrrrrrrras_target_processorosz"DonutProcessor.as_target_processorFcCs |dkr|j}i}|rtd|tj}|dkr8q|d}t|}td|d|tj}|}|dkr||d}q|}t|} t|} t| d| |tj} | dk r| d} d| krd| kr|j | d |d } | rt | dkr| d } | ||<ng||<| d D]R} | } | |krn| d d krn| dddkrn| dd} || | q,t ||dkr||d ||<|| |t |d}|ddd kr|g|j |ddd |d Sqt |r |r|gS|S|rgSd|iSdS)zS Convert a (generated) token sequence into an ordered JSON format. Nz rzz(.*?)zZ text_sequence)rZget_added_vocabresearch IGNORECASEgroupescapereplacestrip token2jsonrsplitappendfind)rtokensr(r)outputZ start_tokenkeyZ key_escapedZ end_tokenZstart_token_escapedZend_token_escapedcontentvalueleafrrrr4sN       *   zDonutProcessor.token2jsoncCstdt|jS)Nzg`feature_extractor_class` is deprecated and will be removed in v5. Use `image_processor_class` instead.)r r r image_processor_classr$rrrfeature_extractor_classs z&DonutProcessor.feature_extractor_classcCstdt|jS)Nz[`feature_extractor` is deprecated and will be removed in v5. Use `image_processor` instead.)r r r rr$rrrrs z DonutProcessor.feature_extractor)NN)FN)__name__ __module__ __qualname____doc__ attributesr>Ztokenizer_classrr r!r#rr%r4propertyr?r __classcell__rrrrrs!  4 r)rCr-r contextlibrZprocessing_utilsrrrrrrs