U dV2@sdZddlmZmZddlmZmZddlmZm Z m Z m Z ddl Z GdddeZGdd d eZGd d d eZdd d ZddZGdddeZdS)aH Implementation of an in-memory dataset with structured schema. Use this to store and iterate through datasets with complex schema that fit in memory. Iterating through entries of this dataset is very fast since the dataset is stored as a set of native Caffe2 tensors, thus no type conversion or deserialization is necessary. )core workspace)ReaderWriter)Structfrom_blob_listfrom_column_listInitEmptyRecordNc@s.eZdZd ddZddZddZd d Zd S) _DatasetReaderFcCs<t||||_|p"|jd|_||_||_d|_dS)z7Don't call this directly. Instead, use dataset.reader()_cursorN)r__init__contentdatasetname batch_sizeenforce_batch_sizecursor)selfrrrrr9/tmp/pip-unpacked-wheel-ua33x9lu/caffe2/python/dataset.pyr s z_DatasetReader.__init__cCs,|jdkr(|jg||j|jjd|_dSNfieldsrZCreateTreeCursorZNextScopedBlobrrrrinit_netZexit_netrrrsetup_ex#s   z_DatasetReader.setup_exc Cs|jstd|j}t||jR|j|jg| | |j |j d}t |}||dg|fW5QRSQRXdS)Nsetup not called.)rrr)rAssertionErrorrrr NameScopeNextNamerZ ReadNextBatch field_blobs field_namesrroutput_to_listIsEmptyrZread_netrrrrrread*s  z_DatasetReader.readcCs||jggdSNZ ResetCursorrrnetrrrreset6sz_DatasetReader.resetN)r F)__name__ __module__ __qualname__r rr'r,rrrrr s  r c@s@eZdZdddZddZddZd d Zdd d ZddZd S)_DatasetRandomReaderr FcCsHt||||_d|_|p(|jd|_||_||_||_||_ dS)z>Don't call this directly. Instead, use dataset.random_reader()Nr ) rr rrrrindicesr loop_overr)rrrr1rr2rrrrr ;sz_DatasetRandomReader.__init__cCs,|jdkr(|jg||j|jjd|_dSrrrrrrrGs   z_DatasetRandomReader.setup_excCs||jggdSr(r)r*rrrr,Nsz_DatasetRandomReader.resetcCs2||||jg|jd}||_dS)Noffsets)r,Z ComputeOffsetrrrr"r3)rr+r3rrr computeoffsetQs  z"_DatasetRandomReader.computeoffsetNcCsh|j}d}|r4||ks&td||}|||j|jg|d|||d}||_ dS)NzMust be valid field.r1)sort_by_field_idx shuffle_sizer) rrr#rindexr,ZSortAndShufflerr"r1)rr+Z sort_by_fieldr7rrr6r1rrrsort_and_shuffleXs   z%_DatasetRandomReader.sort_and_shufflec Cs|jstd|jstd|js*td|j}t||j ^|j |j|j|jg| | |j |j|jd}t|}||dg|fW5QRSQRXdS)Nzsetup_ex not calledzsort_and_shuffle not calledzcomputeoffset not called)rrr2r)rrr1r3rrrr r!rZReadRandomBatchr"r#rrr2r$r%r&rrrr'ks   z_DatasetRandomReader.read)r FF)Nr r ) r-r.r/r rr,r4r9r'rrrrr0:s  r0c@s,eZdZddZddZddZddZd S) _DatasetWritercCs||_d|_dS)z7Don't call this directly. Use dataset.writer() instead.N)_contentmutex)rrrrrr }sz_DatasetWriter.__init__cCs|jdkr|g|_dSr()r<Z CreateMutexrrrrrs z_DatasetWriter.setup_excCsz|jdk std|j}t|t|ksDtdt|t|f|j|g|jd||jg|t||dS)a Add operations to `net` that append the blobs in `fields` to the end of the dataset. An additional operator will also be added that checks the consistency of the data in `fields` against the dataset schema. Args: writer_net: The net that will contain the Append operators. fields: A list of BlobReference to be appeneded to this dataset. NrzExpected %s fields, got %s.r) r<rr;r"lenZCheckDatasetConsistencyr#Z AtomicAppendlist)rZ writer_netrr"rrrwrites  z_DatasetWriter.writecCsdS)z+Commit is a no-op for an in-memory dataset.Nr)rZ finish_netrrrcommitsz_DatasetWriter.commitN)r-r.r/r rr?r@rrrrr:|sr:cCsHt|tjstdtj||d}||j|d}t t |||S)a Create a 'constant' by first creating an external input in the given net, and then feeding the corresponding blob with its provided value in the current workspace. The name is automatically generated in order to avoid clashes with existing blob names. z net must be a core.Net instance.)dtype)prefix) isinstancerNetrnparrayZAddExternalInputr!rZFeedBlobstr)r+valuerArZblobrrrConsts rIcCs,td}||ggtj|||dddS)N report_netT)rJZconcurrent_substepsZreport_interval)rrDZPrintZexecution_step)rrZsubstepsZ rows_readrJrrrexecution_step_with_progresss rLc@speZdZdZdddZddZddZd d Zd d Zd dZ ddZ ddZ dddZ dddZ dddZdS)DatasetaSRepresents an in-memory dataset with fixed schema. Use this to store and iterate through datasets with complex schema that fit in memory. Iterating through entries of this dataset is very fast since the dataset is stored as a set of native Caffe2 tensors, thus no type conversion or deserialization is necessary. NcCslt|tst|tstdt|tr.t|}||_||_||_|pNd|_ | rb| nd|_ dS)aCreate an un-initialized dataset with schema provided by `fields`. Before this dataset can be used, it must be initialized, either by `init_empty` or `init_from_dataframe`. Args: fields: either a schema.Struct or a list of field names in a format compatible with the one described in schema.py. name: optional name to prepend to blobs that will store the data. zrrrschemar#r field_typesrZ has_blobsr")rrrrrrr s     zDataset.__init__cCs|j|j|j|j|ddS)z} Trims the contents of this dataset so that the number of records is multiple of the given argument. )r multiple_ofN)Z TrimDatasetr"r)rr+rPrrrtrims z Dataset.trimcCst||j|_dS)zInitialize the blobs for this dataset with empty values. Empty arrays will be immediately fed into the current workspace, and `init_net` will take those blobs as external inputs. N)r rNZ clone_schemar")rrrrr init_emptyszDataset.init_emptycs8t|jtjkstfddt|jD|_dS)zInitialize the blobs for this dataset from a Pandas dataframe. Each column of the dataframe will be immediately fed into the current workspace, and the `net` will take this blobs as external inputs. cs*g|]"\}}t|g|dqS))r)rIZ as_matrixflatten).0colfield dataframer+rr sz/Dataset.init_from_dataframe..N)r=rcolumnsr enumerater")rr+rXrrWrinit_from_dataframes zDataset.init_from_dataframecCs|st|jS)zx Return the list of BlobReference pointing to the blobs that contain the data for this dataset. )rr"rrrr get_blobsszDataset.get_blobscCst|j|jS)zi Return a Record of BlobReferences pointing to the full content of this dataset. )rrNr"r]rrrrszDataset.contentcCs|jS)z0Return the list of field names for this dataset.rr]rrrr#szDataset.field_namescCs|jS)z Return the list of field dtypes for this dataset. If a list of strings, not a schema.Struct, was passed to the constructor, this will return a list of dtype(np.void). )rOr]rrrrO szDataset.field_typesr FcCs4|jstdt||||}|dk r0||d|S)aCreate a Reader object that is used to iterate through the dataset. This will append operations to `init_net` that create a TreeCursor, used to iterate through the data. NOTE: Currently, it is not safe to append to a dataset while reading. Args: init_net: net that will be run once to create the cursor. cursor_name: optional name for the blob containing a pointer to the cursor. batch_size: how many samples to read per iteration. Returns: A _DatasetReader that can be used to create operators that will iterate through the dataset. Dataset not initialized.N)r"rr r)rr cursor_namerrreaderrrrras zDataset.readercCs8|jstdt||||||}|dk r4||d|S)aCreate a Reader object that is used to iterate through the dataset. NOTE: The reader order depends on the order in indices. Args: init_net: net that will be run once to create the cursor. indices: blob of reading order cursor_name: optional name for the blob containing a pointer to the cursor. batch_size: how many samples to read per iteration. loop_over: repeat the dataset indefinitely (in the same order) Returns: A DatasetReader that can be used to create operators that will iterate through the dataset according to indices. r_N)r"rr0r)rrr1r`rr2rrarrr random_reader.s zDataset.random_readercCs2|jstdt|}|dk r.||d|S)aCreate a Writer that can be used to append entries into the dataset. NOTE: Currently, it is not safe to append to a dataset while reading from it. NOTE: Currently implementation of writer is not thread safe. TODO: fixme Args: init_net: net that will be run once in order to create the writer. (currently not used) r_N)r"rr:rr)rrwriterrrrrcHs   zDataset.writer)N)NNr F)NNNr FF)N)r-r.r/__doc__r rQrRr\r^rr#rOrarbrcrrrrrMs$       rM)NN)rdZ caffe2.pythonrrZcaffe2.python.dataiorrZcaffe2.python.schemarrrr ZnumpyrEr r0r:rIrLobjectrMrrrrs!B#