U 9%e9 @sddlZddlZddlZddlmZddlmZmZmZm Z m Z m Z m Z ddl mZmZejeZejedZeeZeZW5QRXGdddZe eZe e eZGdd d e ZGd d d ZGd d d ZGdddZ efe e!e!dddZ"dS)N)Template)AnyCallableDictList NamedTupleOptionalTuple)Encoding Tokenizerzvisualizer-styles.cssc@s8eZdZUeed<eed<eed<eeedddZdS) Annotationstartendlabelr rrcCs||_||_||_dSNr)selfr rrrZ/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/tokenizers/tools/visualizer.py__init__szAnnotation.__init__N)__name__ __module__ __qualname__int__annotations__strrrrrrr s r c@s&eZdZUeeed<eeed<dS) CharStateKeytoken_ixanno_ixN)rrrrrrrrrrrs  rc@sHeZdZUeeed<ddZeddZeddZ e dd d Z d S) CharStatechar_ixcCs||_d|_g|_dSr)r rtokens)rr rrrr'szCharState.__init__cCst|jdkr|jdSdS)Nrlenr!rrrrr-szCharState.token_ixcCst|jdkS)zJ BPE tokenizers can output more than one token for a char r"r$rrr is_multitoken1szCharState.is_multitoken)returncCst|j|jdS)N)rr)rrrr$rrr partition_key8szCharState.partition_keyN) rrrrrrrpropertyrr&rr(rrrrr$s    rc@s eZdZdS)AlignedN)rrrrrrrr*?sr*c@seZdZdZejdejdZdee e e e ge fdddZgdfeee e e ed d d Zeeeeefd d dZeeeeedddZeeeeedddZeeeedddZeeeeeedddZdS)EncodingVisualizera Build an EncodingVisualizer Args: tokenizer (:class:`~tokenizers.Tokenizer`): A tokenizer instance default_to_notebook (:obj:`bool`): Whether to render html output in a notebook by default annotation_converter (:obj:`Callable`, `optional`): An optional (lambda) function that takes an annotation in any format and returns an Annotation object z(.{1})?(unk|oov)(.{1})?)flagsTN) tokenizerdefault_to_notebookannotation_converterc CsZ|rDzddlm}m}Wn*tk rB}z tdW5d}~XYnX||_||_||_dS)NrHTMLdisplayzWe couldn't import IPython utils for html display. Are you running in a notebook? You can also pass `default_to_notebook=False` to get back raw HTML )IPython.core.displayr1r2 ImportError Exceptionr-r.annotation_coverter)rr-r.r/r1r2errrrVszEncodingVisualizer.__init__)text annotationsr.r'c Cs|j}|dk r|}|rVzddlm}m}Wn*tk rT}z tdW5d}~XYnX|jdk rptt|j|}|j |}t |||} |r||| n| SdS)a Build a visualization of the given text Args: text (:obj:`str`): The text to tokenize annotations (:obj:`List[Annotation]`, `optional`): An optional list of annotations of the text. The can either be an annotation class or anything else if you instantiated the visualizer with a converter function default_to_notebook (:obj:`bool`, `optional`, defaults to `False`): If True, will render the html in a notebook. Otherwise returns an html string. Returns: The HTML string if default_to_notebook is False, otherwise (default) returns None and renders the HTML in the notebook Nrr0zeWe couldn't import IPython utils for html display. Are you running in a notebook?) r.r3r1r2r4r5r6listmapr-encoder+_EncodingVisualizer__make_html) rr8r9r.Zfinal_default_to_notebookr1r2r7encodinghtmlrrr__call__ls"  zEncodingVisualizer.__call__)r9r'c Cst|dkriSttdd|}t|}td|}|dkrBd}d}d}d}i}t|D](}d |d |d |d ||<||7}qZ|S) a Generates a color palette for all the labels in a given set of annotations Args: annotations (:obj:`Annotation`): A list of annotations Returns: :obj:`dict`: A dictionary mapping labels to colors in HSL format rcSs|jSr)r)xrrrz;EncodingVisualizer.calculate_label_colors.. @ zhsl(,z%,%)r#setr;rsorted) r9labelsZ num_labelsZh_stepslhcolorsrrrrcalculate_label_colorss    z)EncodingVisualizer.calculate_label_colors)consecutive_chars_listr8r>cCs.|d}|jdkr*|j|j}d|dS|d}|j}|jd}|||}g} i} |jdk r| d|jrz| d|jd r| d n | d tj|j|jdk r| d |j|j| d <n | ddd| d} d} | D]\} }| d| d|d7} qd| d| d|dS)a Converts a list of "consecutive chars" into a single HTML element. Chars are consecutive if they fall under the same word, token and annotation. The CharState class is a named tuple with a "partition_key" method that makes it easy to compare if two chars are consecutive. Args: consecutive_chars_list (:obj:`List[CharState]`): A list of CharStates that have been grouped together text (:obj:`str`): The original text being processed encoding (:class:`~tokenizers.Encoding`): The encoding returned from the tokenizer Returns: :obj:`str`: The HTML span for a set of consecutive chars rNz(r%tokenz multi-tokenz odd-tokenz even-tokenz special-tokenZstokz non-tokenzclass=" "z data-z="z) r r!rappendr&r+unk_token_regexsearchjoinitems)rSr8r>firstZstokenlastr rZ span_textZ css_classesZ data_itemscssdatakeyvalrrrconsecutive_chars_to_htmls4             z,EncodingVisualizer.consecutive_chars_to_html)r8r>r9r'cCsXt|||}|dg}|dj}g}t|}|dj}|dk rp||} | j} || } |d| d| d|ddD]} | j}||kr|tj|||d| g}|dk r|d|dk r||} | j} || } |d| d| d|}| |dkr|| q||tj|||d| g}q||tj|||dt|} | S)Nrz&r%)r8r>rZ) r+%_EncodingVisualizer__make_char_statesrrRrr[rfr(HTMLBody)r8r>r9 char_statesZcurrent_consecutive_charsZ prev_anno_ixspansZlabel_colors_dictZ cur_anno_ixannorcolorcsresrrrZ __make_htmlsb      zEncodingVisualizer.__make_html)r8r9r'cCs@dgt|}t|D]$\}}t|j|jD] }|||<q,q|S)a Args: text (:obj:`str`): The raw text we want to align to annotations (:obj:`AnnotationList`): A (possibly empty) list of annotations Returns: A list of length len(text) whose entry at index i is None if there is no annotation on charachter i or k, the index of the annotation that covers index i where k is with respect to the list of annotations N)r# enumerateranger r)r8r9annotation_maprairrrZ__make_anno_map<s  z"EncodingVisualizer.__make_anno_mapc Cst||}ddtt|D}t|jD]B\}}||}|dk r,|\}} t|| D]} || j|qXq,t|D]\} } | || _qx|S)a For each character in the original text, we emit a tuple representing it's "state": * which token_ix it corresponds to * which word_ix it corresponds to * which annotation_ix it corresponds to Args: text (:obj:`str`): The raw text we want to align to annotations (:obj:`List[Annotation]`): A (possibly empty) list of annotations encoding: (:class:`~tokenizers.Encoding`): The encoding returned from the tokenizer Returns: :obj:`List[CharState]`: A list of CharStates, indicating for each char in the text what it's state is cSsg|] }t|qSr)r).0r rrr jsz9EncodingVisualizer.__make_char_states..N) r+"_EncodingVisualizer__make_anno_maprpr#ror!Ztoken_to_charsr[r) r8r>r9rqrirrUoffsetsr rrsr rrrrZ__make_char_statesQs   z%EncodingVisualizer.__make_char_states)TN)rrr__doc__recompile IGNORECASEr\r boolrrrr rrAnnotationListr@ staticmethodrrRrrr rfr=PartialIntListrvrgrrrrr+Cs<  -CAr+)childrenr'cCsd|}d|d|dS)a[ Generates the full html with css from a list of html spans Args: children (:obj:`List[str]`): A list of strings, assumed to be html elements css_styles (:obj:`str`, `optional`): Optional alternative implementation of the css Returns: :obj:`str`: An HTML string with style markup rYz?
z4
)r^)rZ css_stylesZ children_textrrrrhws   rh)# itertoolsosrystringrtypingrrrrrrr Z tokenizersr r pathdirname__file__r^Z css_filenameopenfreadrbr r}rrrrr*r+rrhrrrrs& $    6