U &%e,ã@sLddlZddlZddlmZmZddlmZmZe d¡Z Gdd„dƒZ dS)éN)ÚOptionalÚUnioné)ÚLanguageFilterÚ ProbingStates%[a-zA-Z]*[€-ÿ]+[a-zA-Z]*[^a-zA-Z€-ÿ]?c@sìeZdZdZejfeddœdd„Zddœdd„Zee e dœd d „ƒZ ee e dœd d „ƒZ e eefed œdd„Zeedœdd„ƒZedœdd„Zee eefedœdd„ƒZee eefedœdd„ƒZee eefedœdd„ƒZdS)Ú CharSetProbergffffffî?N)Ú lang_filterÚreturncCs$tj|_d|_||_t t¡|_dS)NT) rÚ DETECTINGÚ_stateÚactiverÚloggingÚ getLoggerÚ__name__Úlogger)Úselfr©rú`/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/pip/_vendor/chardet/charsetprober.pyÚ__init__,szCharSetProber.__init__)r cCs tj|_dS©N)rr r ©rrrrÚreset2szCharSetProber.resetcCsdSrrrrrrÚ charset_name5szCharSetProber.charset_namecCst‚dSr©ÚNotImplementedErrorrrrrÚlanguage9szCharSetProber.language)Úbyte_strr cCst‚dSrr)rrrrrÚfeed=szCharSetProber.feedcCs|jSr)r rrrrÚstate@szCharSetProber.statecCsdS)NgrrrrrÚget_confidenceDszCharSetProber.get_confidence)Úbufr cCst dd|¡}|S)Ns([-])+ó )ÚreÚsub)r rrrÚfilter_high_byte_onlyGsz#CharSetProber.filter_high_byte_onlycCsZtƒ}t |¡}|D]@}| |dd…¡|dd…}| ¡sJ|dkrJd}| |¡q|S)u7 We define three types of bytes: alphabet: english alphabets [a-zA-Z] international: international characters [€-ÿ] marker: everything else [^a-zA-Z€-ÿ] The input buffer can be thought to contain a series of words delimited by markers. This function works to filter all words that contain at least one international character. All contiguous sequences of markers are replaced by a single space ascii character. This filter applies to all scripts which do not use English characters. Néÿÿÿÿó€r!)Ú bytearrayÚINTERNATIONAL_WORDS_PATTERNÚfindallÚextendÚisalpha)r ÚfilteredÚwordsÚwordÚ last_charrrrÚfilter_international_wordsLs    z(CharSetProber.filter_international_wordscCs’tƒ}d}d}t|ƒ d¡}t|ƒD]R\}}|dkrB|d}d}q$|dkr$||krr|sr| |||…¡| d¡d}q$|sŽ| ||d …¡|S) a[ Returns a copy of ``buf`` that retains only the sequences of English alphabet and high byte characters that are not between <> characters. This filter can be applied to all scripts which contain both English characters and extended ASCII characters, but is currently only used by ``Latin1Prober``. FrÚcó>rósÿ