U -e{'@s<ddlZddlZddlmZmZmZmZmZm Z m Z m Z m Z m Z mZmZmZmZmZmZddlmZGdddejZdZGdddejZGdd d ejZGd d d ejZd d ZddZdddddZ d!ee _"dddddZ#dddddZ$d!ee$_"d#dddddZ%dddddZ&dd d!d"Z'dS)$N)IpcReadOptionsIpcWriteOptions ReadStats WriteStatsMessage MessageReaderRecordBatchReader_ReadPandasMixinMetadataVersion read_messageread_record_batch read_schema read_tensor write_tensorget_record_batch_sizeget_tensor_sizec@s eZdZdZdddddZdS)RecordBatchStreamReadera Reader for the Arrow streaming binary format. Parameters ---------- source : bytes/buffer-like, pyarrow.NativeFile, or file-like Python object Either an in-memory buffer, or a readable file object. If you want to use memory map use MemoryMappedFile as source. options : pyarrow.ipc.IpcReadOptions Options for IPC deserialization. If None, default values will be used. memory_pool : MemoryPool, default None If None, default memory pool is used. Noptions memory_poolcCst|}|j|||ddS)Nr _ensure_default_ipc_read_options_open)selfsourcerrrL/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/pyarrow/ipc.py__init__2sz RecordBatchStreamReader.__init____name__ __module__ __qualname____doc__rrrrrr"sraNParameters ---------- sink : str, pyarrow.NativeFile, or file-like Python object Either a file path, or a writable file object. schema : pyarrow.Schema The Arrow schema for data to be written to the file. use_legacy_format : bool, default None Deprecated in favor of setting options. Cannot be provided with options. If None, False will be used unless this default is overridden by setting the environment variable ARROW_PRE_0_15_IPC_FORMAT=1 options : pyarrow.ipc.IpcWriteOptions Options for IPC serialization. If None, default values will be used: the legacy format will not be used unless overridden by setting the environment variable ARROW_PRE_0_15_IPC_FORMAT=1, and the V5 metadata version will be used unless overridden by setting the environment variable ARROW_PRE_1_0_METADATA_VERSION=1.c@s&eZdZdeZdddddZdS)RecordBatchStreamWriterz0Writer for the Arrow streaming binary format {}Nuse_legacy_formatrcCst||}|j|||ddSNr_get_legacy_format_defaultrrsinkschemar%rrrrrSs z RecordBatchStreamWriter.__init__rr r!format_ipc_writer_class_docr"rrrrrr#Nsr#c@s"eZdZdZddddddZdS)RecordBatchFileReadera Class for reading Arrow record batch data from the Arrow binary file format Parameters ---------- source : bytes/buffer-like, pyarrow.NativeFile, or file-like Python object Either an in-memory buffer, or a readable file object. If you want to use memory map use MemoryMappedFile as source. footer_offset : int, default None If the file is embedded in some larger file, this is the byte offset to the very end of the file data options : pyarrow.ipc.IpcReadOptions Options for IPC serialization. If None, default values will be used. memory_pool : MemoryPool, default None If None, default memory pool is used. NrcCst|}|j||||ddS)N footer_offsetrrr)rrr2rrrrrrks zRecordBatchFileReader.__init__)Nrrrrrr0Xsr0c@s&eZdZdeZdddddZdS)RecordBatchFileWriterz1Writer to create the Arrow binary file format {}Nr$cCst||}|j|||ddSr&r(r*rrrrxs zRecordBatchFileWriter.__init__r-rrrrr3rsr3cCs|dk r|dk rtdn$|r>t|ts:tdt||Stj}|dkrbtt t j dd}tt t j ddr~tj }t||dS)Nz8Can provide at most one of options and use_legacy_formatz expected IpcWriteOptions, got {}ZARROW_PRE_0_15_IPC_FORMAT0ZARROW_PRE_1_0_METADATA_VERSION)r%metadata_version) ValueError isinstancer TypeErrorr.typer ZV5boolintosenvirongetZV4)r%rr5rrrr)}s& r)cCs*|r t|ts tdt||p(tS)Nzexpected IpcReadOptions, got {})r7rr8r.r9r'rrrrs  rr$cCst||||dSNr$)r#r+r,r%rrrr new_streamsrAzCreate an Arrow columnar IPC stream writer instance {} Returns ------- writer : RecordBatchStreamWriter A writer for the given sink rcCst|||dS)a Create reader for Arrow streaming format. Parameters ---------- source : bytes/buffer-like, pyarrow.NativeFile, or file-like Python object Either an in-memory buffer, or a readable file object. options : pyarrow.ipc.IpcReadOptions Options for IPC serialization. If None, default values will be used. memory_pool : MemoryPool, default None If None, default memory pool is used. Returns ------- reader : RecordBatchStreamReader A reader for the given source r)r)rrrrrr open_streamsrBcCst||||dSr?)r3r@rrrnew_filesrCzCreate an Arrow columnar IPC file writer instance {} Returns ------- writer : RecordBatchFileWriter A writer for the given sink cCst||||dS)a Create reader for Arrow file format. Parameters ---------- source : bytes/buffer-like, pyarrow.NativeFile, or file-like Python object Either an in-memory buffer, or a readable file object. footer_offset : int, default None If the file is embedded in some larger file, this is the byte offset to the very end of the file data. options : pyarrow.ipc.IpcReadOptions Options for IPC serialization. If None, default values will be used. memory_pool : MemoryPool, default None If None, default memory pool is used. Returns ------- reader : RecordBatchFileReader A reader for the given source r1)r0)rr2rrrrr open_files rDnthreadspreserve_indexc CsFtjj|||d}t}t||j}||W5QRX|S)a Serialize a pandas DataFrame into a buffer protocol compatible object. Parameters ---------- df : pandas.DataFrame nthreads : int, default None Number of threads to use for conversion to Arrow, default all CPUs. preserve_index : bool, default None The default of None will store the index as a column, except for RangeIndex which is stored as metadata only. If True, always preserve the pandas index data as a column. If False, no index information is saved and the result will have a default RangeIndex. Returns ------- buf : buffer An object compatible with the buffer protocol. rE)paZ RecordBatchZ from_pandasZBufferOutputStreamr#r,Z write_batchgetvalue)ZdfrFrGbatchr+writerrrrserialize_pandass rLT use_threadsc Cs4t|}t|}|}W5QRX|j|dS)aDeserialize a buffer protocol compatible object into a pandas DataFrame. Parameters ---------- buf : buffer An object compatible with the buffer protocol. use_threads : bool, default True Whether to parallelize the conversion using multiple threads. Returns ------- df : pandas.DataFrame The buffer deserialized as pandas DataFrame rM)rHZ BufferReaderrZread_allZ to_pandas)bufrNZ buffer_readerreadertablerrrdeserialize_pandas s  rR)N)(r<ZpyarrowrHZ pyarrow.librrrrrrrr r r r r rrrrlibZ_RecordBatchStreamReaderrr/Z_RecordBatchStreamWriterr#Z_RecordBatchFileReaderr0Z_RecordBatchFileWriterr3r)rrAr.r"rBrCrDrLrRrrrrs,H