U ,-e3$@shdZddlZddlmZmZmZddlmZddlm Z m Z m Z m Z m Z ddlmZGdd d eZdS) z! Processor class for LayoutLMv2. N)ListOptionalUnion)ProcessorMixin) BatchEncodingPaddingStrategyPreTokenizedInput TextInputTruncationStrategy) TensorTypecs*eZdZdZddgZdZdZdfdd Zdee e e e e e fe ee e e fee e e e e e e fe ee e e e e feeeeefeeeefe e e e e e ee eeeeeee eeefed d dZddZddZddZeddZeddZeddZZS)LayoutLMv2Processora` Constructs a LayoutLMv2 processor which combines a LayoutLMv2 image processor and a LayoutLMv2 tokenizer into a single processor. [`LayoutLMv2Processor`] offers all the functionalities you need to prepare data for the model. It first uses [`LayoutLMv2ImageProcessor`] to resize document images to a fixed size, and optionally applies OCR to get words and normalized bounding boxes. These are then provided to [`LayoutLMv2Tokenizer`] or [`LayoutLMv2TokenizerFast`], which turns the words and bounding boxes into token-level `input_ids`, `attention_mask`, `token_type_ids`, `bbox`. Optionally, one can provide integer `word_labels`, which are turned into token-level `labels` for token classification tasks (such as FUNSD, CORD). Args: image_processor (`LayoutLMv2ImageProcessor`): An instance of [`LayoutLMv2ImageProcessor`]. The image processor is a required input. tokenizer (`LayoutLMv2Tokenizer` or `LayoutLMv2TokenizerFast`): An instance of [`LayoutLMv2Tokenizer`] or [`LayoutLMv2TokenizerFast`]. The tokenizer is a required input. image_processor tokenizerZLayoutLMv2ImageProcessor)ZLayoutLMv2TokenizerZLayoutLMv2TokenizerFastNc sdd}d|kr"tdt|d}|dk r.|n|}|dkrBtd|dkrRtdt||dS)Nfeature_extractorzhThe `feature_extractor` argument is deprecated and will be removed in v5, use `image_processor` instead.z)You need to specify an `image_processor`.z"You need to specify a `tokenizer`.)warningswarn FutureWarningpop ValueErrorsuper__init__)selfrrkwargsr __class__u/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/models/layoutlmv2/processing_layoutlmv2.pyr2s zLayoutLMv2Processor.__init__TFr)text text_pairboxes word_labelsadd_special_tokenspadding truncation max_lengthstridepad_to_multiple_ofreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverbosereturn_tensorsreturncKs|jjr|dk rtd|jjr0|dk r0td|dkrH|dkrHtd|j||d}|dk r|jjr|dkrt|tr~|g}|d}|jf|dk r|n|d|dk r|nd|dk r|n|d ||||| | | | | ||||||d |}|d }|dkr |||d }||d <|S)a This method first forwards the `images` argument to [`~LayoutLMv2ImageProcessor.__call__`]. In case [`LayoutLMv2ImageProcessor`] was initialized with `apply_ocr` set to `True`, it passes the obtained words and bounding boxes along with the additional arguments to [`~LayoutLMv2Tokenizer.__call__`] and returns the output, together with resized `images`. In case [`LayoutLMv2ImageProcessor`] was initialized with `apply_ocr` set to `False`, it passes the words (`text`/``text_pair`) and `boxes` specified by the user along with the additional arguments to [`~LayoutLMv2Tokenizer.__call__`] and returns the output, together with resized `images``. Please refer to the docstring of the above two methods for more information. NzdYou cannot provide bounding boxes if you initialized the image processor with apply_ocr set to True.zaYou cannot provide word labels if you initialized the image processor with apply_ocr set to True.TFzKYou cannot return overflowing tokens without returning the offsets mapping.)imagesr/wordsr )rrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/Z pixel_valuesoverflow_to_sample_mappingimage)rZ apply_ocrr isinstancestrrrget_overflowing_images)rr1rrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/rfeaturesZencoded_inputsrrr__call__DsT"   zLayoutLMv2Processor.__call__cCsLg}|D]}|||qt|t|krHtdt|dt||S)Nz`Expected length of images to be the same as the length of `overflow_to_sample_mapping`, but got z and )appendlenr)rr1r3Zimages_with_overflowZ sample_idxrrrr7sz*LayoutLMv2Processor.get_overflowing_imagescOs|jj||S)z This method forwards all its arguments to PreTrainedTokenizer's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information. )r batch_decoderargsrrrrr<sz LayoutLMv2Processor.batch_decodecOs|jj||S)z This method forwards all its arguments to PreTrainedTokenizer's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. )rdecoder=rrrr?szLayoutLMv2Processor.decodecCsdddddgS)NZ input_idsZbboxZtoken_type_idsZattention_maskr4rrrrrmodel_input_namessz%LayoutLMv2Processor.model_input_namescCstdt|jS)Nzg`feature_extractor_class` is deprecated and will be removed in v5. Use `image_processor_class` instead.)rrrimage_processor_classr@rrrfeature_extractor_classs z+LayoutLMv2Processor.feature_extractor_classcCstdt|jS)Nz[`feature_extractor` is deprecated and will be removed in v5. Use `image_processor` instead.)rrrrr@rrrrs z%LayoutLMv2Processor.feature_extractor)NN)NNNNTFFNrNNNFFFFTN)__name__ __module__ __qualname____doc__ attributesrBZtokenizer_classrrr r rrintboolr6rr r rr9r7r<r?propertyrArCr __classcell__rrrrr sj   V  r )rGrtypingrrrZprocessing_utilsrZtokenization_utils_baserrr r r utilsr r rrrrs