U ,È-eËã@sTdZddlmZmZddlmZddlmZddlm Z m Z m Z Gdd„deƒZ d S) z Processor class for MarkupLM. é)ÚOptionalÚUnioné)Ú TensorType)ÚProcessorMixin)Ú BatchEncodingÚPaddingStrategyÚTruncationStrategyc@s”eZdZdZdZdZdZdeeee e feee e fe e e e e e ee eeeeeee ee efedœd d „Zd d „Zd d„Zedd„ƒZdS)ÚMarkupLMProcessoraJ Constructs a MarkupLM processor which combines a MarkupLM feature extractor and a MarkupLM tokenizer into a single processor. [`MarkupLMProcessor`] offers all the functionalities you need to prepare data for the model. It first uses [`MarkupLMFeatureExtractor`] to extract nodes and corresponding xpaths from one or more HTML strings. Next, these are provided to [`MarkupLMTokenizer`] or [`MarkupLMTokenizerFast`], which turns them into token-level `input_ids`, `attention_mask`, `token_type_ids`, `xpath_tags_seq` and `xpath_subs_seq`. Args: feature_extractor (`MarkupLMFeatureExtractor`): An instance of [`MarkupLMFeatureExtractor`]. The feature extractor is a required input. tokenizer (`MarkupLMTokenizer` or `MarkupLMTokenizerFast`): An instance of [`MarkupLMTokenizer`] or [`MarkupLMTokenizerFast`]. The tokenizer is a required input. parse_html (`bool`, *optional*, defaults to `True`): Whether or not to use `MarkupLMFeatureExtractor` to parse HTML strings into nodes and corresponding xpaths. ZMarkupLMFeatureExtractor)ZMarkupLMTokenizerZMarkupLMTokenizerFastTNFr)Úadd_special_tokensÚpaddingÚ truncationÚ max_lengthÚstrideÚpad_to_multiple_ofÚreturn_token_type_idsÚreturn_attention_maskÚreturn_overflowing_tokensÚreturn_special_tokens_maskÚreturn_offsets_mappingÚ return_lengthÚverboseÚreturn_tensorsÚreturncKsê|jrR|dkrtdƒ‚|dk s.|dk s.|dk r6tdƒ‚| |¡}|d}|d}n(|dk rbtdƒ‚|dksr|dkrztdƒ‚|dk r˜|jr˜t|tƒr˜|g}|jf|dk rª|n||dk r¸|nd|||||| | | | | ||||||dœ|—Ž}|S) aÆ This method first forwards the `html_strings` argument to [`~MarkupLMFeatureExtractor.__call__`]. Next, it passes the `nodes` and `xpaths` along with the additional arguments to [`~MarkupLMTokenizer.__call__`] and returns the output. Optionally, one can also provide a `text` argument which is passed along as first sequence. Please refer to the docstring of the above two methods for more information. NzDMake sure to pass HTML strings in case `parse_html` is set to `True`zUPlease don't pass nodes, xpaths nor node labels in case `parse_html` is set to `True`ÚnodesÚxpathsz@You have passed HTML strings but `parse_html` is set to `False`.zIMake sure to pass nodes and xpaths in case `parse_html` is set to `False`)ÚtextZ text_pairrÚ node_labelsr r r rrrrrrrrrrr)Ú parse_htmlÚ ValueErrorZfeature_extractorÚ isinstanceÚstrÚ tokenizer)ÚselfZ html_stringsrrrZ questionsr r r rrrrrrrrrrrÚkwargsÚfeaturesZencoded_inputs©r&úq/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/models/markuplm/processing_markuplm.pyÚ__call__0sP!ÿ   îízMarkupLMProcessor.__call__cOs|jj||ŽS)z¾ This method forwards all its arguments to TrOCRTokenizer's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information. )r"Ú batch_decode©r#Úargsr$r&r&r'r)€szMarkupLMProcessor.batch_decodecOs|jj||ŽS)z¸ This method forwards all its arguments to TrOCRTokenizer's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. )r"Údecoder*r&r&r'r,‡szMarkupLMProcessor.decodecCs |jj}|S)N)r"Úmodel_input_names)r#Ztokenizer_input_namesr&r&r'r-Žsz#MarkupLMProcessor.model_input_names)NNNNNTFNNrNNNFFFFTN)Ú__name__Ú __module__Ú __qualname__Ú__doc__Zfeature_extractor_classZtokenizer_classrÚboolrr!rr rÚintrrr(r)r,Úpropertyr-r&r&r&r'r sXì  ê Pr N) r1ÚtypingrrZ file_utilsrZprocessing_utilsrZtokenization_utils_baserrr r r&r&r&r'Ús