U sVcL<@sddlmZddlmZddlmZmZmZmZddl Z ddl Z ddl m mZddlmZmZmZmZddlmZddlmZddlmZmZdd lmZdd lmZdd lm Z dd l!m"Z"m#Z#erdd l$m%Z%m&Z&Gddde"Z'dddddZ(dddddZ)dS)) annotations) defaultdict) TYPE_CHECKINGHashableMappingSequenceN) ArrayLikeDtypeArgDtypeObj ReadCsvBuffer) DtypeWarning)find_stack_level)is_categorical_dtype pandas_dtype)union_categoricals)ExtensionDtype)ensure_index_from_sequences) ParserBase is_index_col)Index MultiIndexcseZdZUded<ded<dddfdd Zdd d d Zdd d dZd dddddZdddddZddZ d!dddddZ Z S)"CParserWrapperbool low_memoryzparsers.TextReader_readerzReadCsvBuffer[str]None)srcreturnc st||_|}|dd_jdk |d<j|d<jj |d<dD]}||dqPt | dd|d<t j |f|_jj_jdk}jjdkrd_n"jjj|\___}jdkrjrfdd tjjD_nttjj_jdd_jrȈjjjdk sLtjd krxtjsxjt jt krfd d t!jD_t jt krȈj"j#j_j$snjj%d kr8t&jr8d _'(jj\}__jdkr8|_jjdkrn|snjdk s\tdgt j_jj%d k_)dS)NrFZallow_leading_colsusecols on_bad_lines)Zstorage_optionsencodingZ memory_map compressionZerror_bad_linesZwarn_bad_linesdtypecsg|]}j|qS)prefix).0iselfr#F/tmp/pip-unpacked-wheel-xj8nt62q/pandas/io/parsers/c_parser_wrapper.py jsz+CParserWrapper.__init__..stringcs$g|]\}}|ks|kr|qSr#r#)r%r&nrr#r)r*srT)*super__init__kwdscopypopr index_colrrvalueensure_dtype_objsgetparsersZ TextReaderrZ unnamed_colsnamesheaderZ_extract_multi_indexer_columns index_names col_namesr$rangeZ table_widthlist orig_names_evaluate_usecolsAssertionErrorZ usecols_dtypesetissubsetZ_validate_usecols_nameslen enumerateZ_validate_parse_dates_presence_set_noconvert_columns_has_complex_date_col leading_colsrZ_name_processed_clean_index_namesZ_implicit_index)r(rr0keyZ passed_namesr: __class__)r(rr)r/1s               zCParserWrapper.__init__)rcCs(z|jWntk r"YnXdSN)rclose ValueErrorr'r#r#r)rMszCParserWrapper.closecs^|jdk stddt|jDfdd|jD}|||j}|D]}|j|qHdS)z Set the columns that should not undergo dtype conversions. Currently, any column that is involved with date parsing will not undergo such conversions. NcSsi|]\}}||qSr#r#)r%r&xr#r#r) sz9CParserWrapper._set_noconvert_columns..csg|] }|qSr#r#r%rOZ names_dictr#r)r*sz9CParserWrapper._set_noconvert_columns..)r>r@rDr8Z_set_noconvert_dtype_columnsrZ set_noconvert)r(Z col_indicesZnoconvert_columnscolr#rRr)rEsz%CParserWrapper._set_noconvert_columnsNz int | Nonez_tuple[Index | MultiIndex | None, Sequence[Hashable] | MultiIndex, Mapping[Hashable, ArrayLike]])nrowsrc s`z,|jr|j|}t|}n |j|}Wntk r|jrd|_||j}|j ||j |j |j dd\}}||j|jdk r|fdd|D}||fYS|YnXd|_|j}|jjr|jrtdg}t|jjD]H}|j dkr||} n||j |} |j| |dd} || qt|}|jdk rj||}||}t|} d dt|| D}|||\} } || |j} nt|} |jdk st t!|j}||}|jdk r||}d d | D} |jdkr |"|| d dt|| D}|||\}} |#| | |\}} || | fS) NFr"r"csi|]\}}|kr||qSr#r#)r%kvcolumnsr#r)rPsz'CParserWrapper.read..z file structure not yet supportedT)try_parse_datescSsi|]\}\}}||qSr#r#r%rVr&rWr#r#r)rP"s cSsg|] }|dqS)r#rQr#r#r)r*:sz'CParserWrapper.read..cSsi|]\}\}}||qSr#r#r[r#r#r)rP>s )$rrZread_low_memory_concatenate_chunksread StopIterationZ _first_chunkZ_maybe_dedup_namesr>Z_get_empty_metar3r:r0r6Z_maybe_make_multi_index_columnsr;r_filter_usecolsitemsrMr8rGrFNotImplementedErrorr<r2_maybe_parse_datesappendrsortedzipZ_do_date_conversionsr@r=Z_check_data_lengthZ _make_index)r(rTchunksdatar8indexZcol_dictZarraysr&valuesZ data_tupsZ column_namesZ date_dataZalldatar#rXr)r^st                     zCParserWrapper.readzSequence[Hashable])r8rcs@||j|dk r.)r?rrCrD)r(r8r#r-r)r`Es  zCParserWrapper._filter_usecolscCsHt|jjd}d}|jjdkr@|jdk r@|||j\}}|_||fS)Nr)r=rr9rGr3rH)r(r8Z idx_namesr#r#r)_get_index_namesNs zCParserWrapper._get_index_namesTint)rirZcCs|r||r||}|SrL)Z_should_parse_datesZ _date_conv)r(rjrirZr#r#r)rcYs z!CParserWrapper._maybe_parse_dates)N)T) __name__ __module__ __qualname____annotations__r/rMrEr^r`rlrc __classcell__r#r#rJr)r-s k  rzlist[dict[int, ArrayLike]]dict)rgrc st|d}g}i}|D]fdd|D}dd|D}dd|D}t|dkrt|g}|ttkr|t| }t |rt |dd |<qt |t r|} | ||<qt||<q|r d |} d d | d g} tj| ttd|S)z Concatenate chunks of data read with low_memory=True. The tricky part is handling Categoricals, where different chunks may have different inferred categories. rcsg|]}|qSr#)r2)r%chunkrkr#r)r*ksz'_concatenate_chunks..cSsh|] }|jqSr#rU)r%ar#r#r) msz&_concatenate_chunks..cSsh|]}t|s|qSr#)rrQr#r#r)rwosr\F)Zsort_categories, z Columns (zK) have mixed types. Specify dtype option on import or set low_memory=False.) stacklevel)r=keysrCnpZfind_common_typer"objectrdstrr2rr isinstancerZconstruct_array_typeZ_concat_same_typeZ concatenatejoinwarningswarnr r ) rgr8Zwarning_columnsresultZarrsZdtypesZ numpy_dtypesZ common_typer"Z array_typeZ warning_namesZwarning_messager#rur)r]_s>     r]z*DtypeArg | dict[Hashable, DtypeArg] | Nonez*DtypeObj | dict[Hashable, DtypeObj] | None)r"rcsxttrHttfdd}D]}t|||<q.|SttrdfddDSdk rttSS)zc Ensure we have either None, a dtype object, or a dictionary mapping to dtype objects. csSrLr#r#) default_dtyper#r)z#ensure_dtype_objs..csi|]}|t|qSr#)r)r%rVrUr#r)rPsz%ensure_dtype_objs..N)rrrdefault_factoryr{rs)r"Zdtype_convertedrIr#)rr"r)r5s    r5)* __future__r collectionsrtypingrrrrrZnumpyr|Zpandas._libs.parsersZ_libsr7Zpandas._typingrr r r Z pandas.errorsr Zpandas.util._exceptionsr Zpandas.core.dtypes.commonrrZpandas.core.dtypes.concatrZpandas.core.dtypes.dtypesrZpandas.core.indexes.apirZpandas.io.parsers.base_parserrrZpandasrrrr]r5r#r#r#r)s(       4A