U
饡c0 @ s d dl mZ d dlZd dlmZ d dlmZ d dlmZ G dd dZ dZ
G dd d Z dd
d Z d e _
d
ddddddddddd ddZde
e_
dS ) )IntegralN)Table) _resolve_filesystem_and_pathc @ s e Zd ZdZdd Zedd Zedd Zedd Zed
d Z ed d
Z
edd Z edd Z edd Z
edd Zedd Zedd Zedd Zed d Zed d Zed d! Zed"d# Zed$d% Zd-d'd(Zd.d)d*Zd/d+d,Zd&S )0ORCFilea
Reader interface for a single ORC file
Parameters
----------
source : str or pyarrow.NativeFile
Readable source. For passing Python file objects or byte buffers,
see pyarrow.io.PythonFileInterface or pyarrow.io.BufferReader.
c C s t | _| j| d S N)_orcZ ORCReaderreaderopen)selfsource r //tmp/pip-unpacked-wheel-qoi2rb4q/pyarrow/orc.py__init__& s
zORCFile.__init__c C s
| j S )z/The file metadata, as an arrow KeyValueMetadata)r metadatar
r r r
r * s zORCFile.metadatac C s
| j S )z#The file schema, as an arrow schema)r schemar r r r
r / s zORCFile.schemac C s
| j S )z The number of rows in the file)r nrowsr r r r
r 4 s z
ORCFile.nrowsc C s
| j S )z!The number of stripes in the file)r nstripesr r r r
r 9 s zORCFile.nstripesc C s
| j S )z4Format version of the ORC file, must be 0.11 or 0.12)r file_versionr r r r
r > s zORCFile.file_versionc C s
| j S )z2Software instance and version that wrote this file)r software_versionr r r r
r C s zORCFile.software_versionc C s
| j S )z Compression codec of the file)r compressionr r r r
r H s zORCFile.compressionc C s
| j S )z?Number of bytes to buffer for the compression codec in the file)r compression_sizer r r r
r M s zORCFile.compression_sizec C s
| j S )z{Name of the writer that wrote this file.
If the writer is unknown then its Writer ID
(a number) is returned)r writerr r r r
r R s zORCFile.writerc C s
| j S )zVersion of the writer)r writer_versionr r r r
r Y s zORCFile.writer_versionc C s
| j S )zRNumber of rows per an entry in the row index or 0
if there is no row index)r row_index_strider r r r
r ^ s zORCFile.row_index_stridec C s
| j S )zNumber of stripe statistics)r nstripe_statisticsr r r r
r d s zORCFile.nstripe_statisticsc C s
| j S )z/Length of the data stripes in the file in bytes)r content_lengthr r r r
r i s zORCFile.content_lengthc C s
| j S )z`_
is the older version
while `Hive 0.12 / ORC v1 `_
is the newer one.
batch_size : int, default 1024
Number of rows the ORC writer writes at a time.
stripe_size : int, default 64 * 1024 * 1024
Size of each ORC stripe in bytes.
compression : string, default 'uncompressed'
The compression codec.
Valid values: {'UNCOMPRESSED', 'SNAPPY', 'ZLIB', 'LZ4', 'ZSTD'}
Note that LZ0 is currently not supported.
compression_block_size : int, default 64 * 1024
Size of each compression block in bytes.
compression_strategy : string, default 'speed'
The compression strategy i.e. speed vs size reduction.
Valid values: {'SPEED', 'COMPRESSION'}
row_index_stride : int, default 10000
The row index stride i.e. the number of rows per
an entry in the row index.
padding_tolerance : double, default 0.0
The padding tolerance.
dictionary_key_size_threshold : double, default 0.0
The dictionary key size threshold. 0 to disable dictionary encoding.
1 to always enable dictionary encoding.
bloom_filter_columns : None, set-like or list-like, default None
Columns that use the bloom filter.
bloom_filter_fpp : double, default 0.05
Upper limit of the false-positive rate of the bloom filter.
c @ sd e Zd ZdeZdZddddddd d
d
d d d
ddZdd Zdd Z dd Z
dd Z dd Z d S ) ORCWritera
Writer interface for a single ORC file
Parameters
----------
where : str or pyarrow.io.NativeFile
Writable target. For passing Python file objects or byte buffers,
see pyarrow.io.PythonFileInterface, pyarrow.io.BufferOutputStream
or pyarrow.io.FixedSizeBufferWriter.
{}
F0.12 uncompressed speed' N皙? r
batch_size stripe_sizer compression_block_sizecompression_strategyr padding_tolerance dictionary_key_size_thresholdbloom_filter_columnsbloom_filter_fppc
C s8 t | _| jj||||||||| |
| | d d| _d S )Nr> T)r r4 r r is_open)
r
wherer r? r@ r rA rB r rC rD rE rF r r r
r s
zORCWriter.__init__c C s | d S r closer r r r
__del__ s zORCWriter.__del__c C s | S r r r r r r
__enter__ s zORCWriter.__enter__c O s | d S r rI )r
argskwargsr r r
__exit__ s zORCWriter.__exit__c C s | j s
t| j| dS )a
Write the table into an ORC file. The schema of the table must
be equal to the schema used when opening the ORC file.
Parameters
----------
table : pyarrow.Table
The table to be written into the ORC file
N)rG AssertionErrorr write)r
tabler r r
rQ s
zORCWriter.writec C s | j r| j d| _ dS )z$
Close the ORC file
FN)rG r rJ r r r r
rJ " s
zORCWriter.close)
r/ r0 r1 format_orc_writer_args_docsr2 rG r rK rL rO rQ rJ r r r r
r4 s*
r4 c C s\ t | |\}}|d k r ||} |d k rHt|dkrHt| |}nt| j|d}|S )Nr r+ )r Zopen_input_filer# r r. select)r r'
filesystempathresultr r r
read_table+ s
rY aI
Read a Table from an ORC file.
Parameters
----------
source : str, pyarrow.NativeFile, or file-like object
If a string passed, can be a single file name. For file-like objects,
only read a single file. Use pyarrow.BufferReader to read a file
contained in a bytes or buffer-like object.
columns : list
If not None, only these columns will be read from the file. A column
name may be a prefix of a nested field, e.g. 'a' will select 'a.b',
'a.c', and 'a.d.e'. If empty, no columns will be read. Note
that the table will still have the correct num_rows set despite having
no columns.
filesystem : FileSystem, default None
If nothing passed, will be inferred based on path.
Path will try to be found in the local on-disk filesystem otherwise
it will be parsed as an URI to determine the filesystem.
r5 r6 r7 r8 r9 r: r; r< r= r> c C s^ t |tr$tjdtdd || } }t||||||||| |
| | d }
|
| W 5 Q R X d S )NzThe order of the arguments has changed. Pass as 'write_table(table, where)' instead. The old order will raise an error in the future. )
stacklevelr> )r! r warningswarn
FutureWarningr4 rQ )rR rH r r? r@ r rA rB r rC rD rE rF r r r r
write_tableN s.
r_ a]
Write a table into an ORC file.
Parameters
----------
table : pyarrow.lib.Table
The table to be written into the ORC file
where : str or pyarrow.io.NativeFile
Writable target. For passing Python file objects or byte buffers,
see pyarrow.io.PythonFileInterface, pyarrow.io.BufferOutputStream
or pyarrow.io.FixedSizeBufferWriter.
{}
)NN)Znumbersr r\ Z pyarrow.libr Z pyarrow._orcr Z
pyarrow.fsr r rT r4 rY r2 r_ rS r r r r
s2 #"L
$