U -eI @sUddlmZddlmZddlmZmZmZddlZ ddl Z ddl m Z ddlmZeZeZeZeje e e e deje e e e deje e e dej de iej!de "iiZ#d e$d <d6d d dddZ%d7d dddZ&d8d dddddZ'd9ddddddZ(d:ddddddZ)d;ddd dd!d"Z*d#d$Z+d%d&Z,dd-d.d)d(d(dd/d3d4d5Z/dS)?) annotations)Any) DtypeKind ColumnBuffersColumnNullTypeN)Dtype) @)r r r rzdict[DtypeKind, dict[int, Any]]_PYARROW_DTYPESTDataFrameObjectzpa.Table)dfreturncCsPt|tjr|St|tjr*tj|gSt|ds>> import pyarrow >>> from pyarrow.interchange import from_dataframe Convert a pandas dataframe to a pyarrow table: >>> import pandas as pd >>> df = pd.DataFrame({ ... "n_atendees": [100, 10, 1], ... "country": ["Italy", "Spain", "Slovenia"], ... }) >>> df n_atendees country 0 100 Italy 1 10 Spain 2 1 Slovenia >>> from_dataframe(df) pyarrow.Table n_atendees: int64 country: large_string ---- n_atendees: [[100,10,1]] country: [["Italy","Spain","Slovenia"]] __dataframe__z#`df` does not support __dataframe__) allow_copy) isinstancepaTable RecordBatch from_batcheshasattr ValueError_from_dataframer)rrrc/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/pyarrow/interchange/from_dataframe.pyfrom_dataframe=s*    r)rcCs2g}|D]}t||}||q tj|S)a Build a ``pa.Table`` from the DataFrame interchange object. Parameters ---------- df : DataFrameObject Object supporting the interchange protocol, i.e. `__dataframe__` method. allow_copy : bool, default: True Whether to allow copying the memory to perform the conversion (if false then zero-copy approach is requested). Returns ------- pa.Table )Z get_chunksprotocol_df_chunk_to_pyarrowappendrrr)rrZbatcheschunkbatchrrrrss    rboolzpa.RecordBatch)rrrcCsi}|D]}t|ts*td|d||krBtd|d||}|jd}|tjtjtj tj tj fkrt ||||<q |tj krt||||<q |tjkrt||||<q td|dq tj|S)a Convert interchange protocol chunk to ``pa.RecordBatch``. Parameters ---------- df : DataFrameObject Object supporting the interchange protocol, i.e. `__dataframe__` method. allow_copy : bool, default: True Whether to allow copying the memory to perform the conversion (if false then zero-copy approach is requested). Returns ------- pa.RecordBatch zColumn z is not a stringz is not uniquerz Data type z not handled yet)Z column_namesrstrrZget_column_by_namedtyperINTUINTFLOATSTRINGDATETIMEcolumn_to_arrayBOOLbool_column_to_arrayZ CATEGORICAL categorical_column_to_dictionaryNotImplementedErrorrrZ from_pydict)rrcolumnsnamecolr#rrrrs,      r ColumnObjectzpa.Array)r0rrcCs$|}t|||j|j|}|S)a Convert a column holding one of the primitive dtypes to a PyArrow array. A primitive type is one of: int, uint, float, bool (1 bit). Parameters ---------- col : ColumnObject allow_copy : bool, default: True Whether to allow copying the memory to perform the conversion (if false then zero-copy approach is requested). Returns ------- pa.Array ) get_buffersbuffers_to_arraysize describe_nulloffsetr0rbuffersdatarrrr)s r)cCs>|s td|}t|||j|j}t|t }|S)aD Convert a column holding boolean dtype to a PyArrow array. Parameters ---------- col : ColumnObject allow_copy : bool, default: True Whether to allow copying the memory to perform the conversion (if false then zero-copy approach is requested). Returns ------- pa.Array zfBoolean column will be casted from uint8 and a copy is required which is forbidden by allow_copy=False) RuntimeErrorr2r3r4r5r6pccastrbool_r7rrrr+s r+zpa.DictionaryArraycCsb|s td|j}|ds"td|d}t|}|}t|||j|j}t j ||}|S)aV Convert a column holding categorical data to a pa.DictionaryArray. Parameters ---------- col : ColumnObject allow_copy : bool, default: True Whether to allow copying the memory to perform the conversion (if false then zero-copy approach is requested). Returns ------- pa.DictionaryArray zjCategorical column will be casted from uint8 and a copy is required which is forbidden by allow_copy=FalseZ is_dictionaryz-Non-dictionary categoricals not supported yet categories) r:Zdescribe_categoricalr-r)r2r3r4r5r6rZDictionaryArrayZ from_arrays)r0rZ categoricalZ cat_column dictionaryr8indicesZ dict_arrayrrrr,s$ r,cCsPtd|}|r>|d|d}}|dkr6|d7}||fStd|dS)z4Parse datetime `format_str` to interpret the `data`.zts([smun]):(.*)sz DateTime kind is not supported: N)rematchgroupr-) format_strZtimestamp_metaunittzrrrparse_datetime_format_str#s rJcCs`|\}}}}|tjkr0t|\}}tj||dSt|i|d}|rL|Std|ddS)z+Map column date type to pyarrow date type. )rINzConversion for  is not yet supported.)rr(rJr timestampr getr-) data_typekind bit_widthZf_string_rHrIZpa_dtyperrr map_date_type5s    rRrintr)r8lengthr5r6rrcCs4|d\}}z|d\}}Wntk r4d}YnXz|d\} } Wntk r^d} YnXtj|j|j|d} |rt||||||} nt| |||||} t|} | r| \}}}}tj| j| j| d}|ddkrt}n|dkrt}nt }tj j ||| || g|d }ntj j | || | g|d }|S) a Build a PyArrow array from the passed buffer. Parameters ---------- buffer : ColumnBuffers Dictionary containing tuples of underlying buffers and their associated dtype. length : int The number of values in the array. describe_null: ColumnNullType Null representation the column dtype uses, as a tuple ``(kind, value)`` offset : int, default: 0 Number of elements to offset from the start of the buffer. allow_copy : bool, default: True Whether to allow copying the memory to perform the conversion (if false then zero-copy approach is requested). Returns ------- pa.Array Notes ----- The returned array doesn't own the memory. The caller of this function is responsible for keeping the memory owner object alive as long as the returned PyArrow array is being used. r9ZvalidityNoffsetsbaserBUr r6) TypeErrorrforeign_bufferptrbufsizevalidity_buffer_from_maskvalidity_buffer_nan_sentinelrRZ large_stringstringArray from_buffers)r8rTr5r6rZ data_buffrN validity_buffvalidity_dtypeZ offset_buffZ offset_dtypedata_pa_bufferZvalidity_pa_buff data_dtyperQZoffset_bit_widthZoffset_pa_bufferZ string_typearrayrrrr3Gsh$        r3 BufferObjectrz pa.Buffer)rcrdr5rTr6rrc Cs|\}}|\}} } } |tjks"t|tjkr0dS|tjksL|tjkr|dkrtj|j |j |d} |tjkr|svt dtj j t|d| g|d} t| t} ntj j t|d| g|d} |dkrt| } | dS|tjkr|dkrtj|j |j |dSt|ddS)a Build a PyArrow buffer from the passed mask buffer. Parameters ---------- validity_buff : BufferObject Tuple of underlying validity buffer and associated dtype. validity_dtype : Dtype Dtype description as a tuple ``(kind, bit-width, format string, endianness)``. describe_null : ColumnNullType Null representation the column dtype uses, as a tuple ``(kind, value)`` length : int The number of values in the array. offset : int, default: 0 Number of elements to offset from the start of the buffer. allow_copy : bool, default: True Whether to allow copying the memory to perform the conversion (if false then zero-copy approach is requested). Returns ------- pa.Buffer NrArVYTo create a bitmask a copy of the data is required which is forbidden by allow_copy=FalserYr* null representation is not yet supported.)rr*AssertionErrorr NON_NULLABLEZ USE_BYTEMASKZ USE_BITMASKrr[r\r]r:rarbint8r;r<r=invertr8r-) rcrdr5rTr6r null_kind sentinel_valZ validity_kindrQZbuffmask mask_boolrrrr^sN!      r^)rerNr5rTr6rrcCs|\}}}}t|} |\} } | tjkr|s2td|tjkrZ|dkrZt|d| dn8tjj | |d|g|d} t | } t | } | dSn| tjkr|std|tjkrt}n| }tjj ||d|g|d} t | | }t |}| dS| tjkr nt|ddS) a Build a PyArrow buffer from NaN or sentinel values. Parameters ---------- data_pa_buffer : pa.Buffer PyArrow buffer for the column data. data_type : Dtype Dtype description as a tuple ``(kind, bit-width, format string, endianness)``. describe_null : ColumnNullType Null representation the column dtype uses, as a tuple ``(kind, value)`` length : int The number of values in the array. offset : int, default: 0 Number of elements to offset from the start of the buffer. allow_copy : bool, default: True Whether to allow copying the memory to perform the conversion (if false then zero-copy approach is requested). Returns ------- pa.Buffer rir z with rKNrYrArj)rRrZUSE_NANr:rr&r-rrarbr;is_nanrnr8Z USE_SENTINELr(int64equalrl)rerNr5rTr6rrOrPrQrfrorpZ pyarrow_datarqZsentinel_dtypeZ sentinel_arrrrrrrr_sT!           r_)T)T)T)T)T)T)rT)rT)rT)0 __future__rtypingrZpyarrow.interchange.columnrrrZpyarrowrrDZpyarrow.computeZcomputer;rr r1rhr$rmZint16Zint32rtr%Zuint8Zuint16Zuint32Zuint64r&Zfloat16Zfloat32Zfloat64r*r'r`r __annotations__rrrr)r+r,rJrRr3r^r_rrrrsj      61#,jT