U d5 @sdZddlZddlZddlmZddlmZddlmZddl m Z m Z ddl m Z ddlmZmZmZdd lmZdd lmZdd lmZeeZd Zd dZdYddZe dddddddgZdej_ Gddde ddddgZ!de!j_ Gd d!d!e"Z#Gd"d#d#e#Z$Gd$d%d%e$Z%Gd&d'd'e#Z&Gd(d)d)e#Z'dZd,d-Z(d[d.d/Z)d0d1Z*d2d3Z+d\d5d6Z,d]d8d9Z-Gd:d;d;e"Z.d^dej8j?fej@ej8jAfejBej8jCfejDej8jEfejFej8jGfejHej8jIfejJej8jKfejLej8jMfg ZNdMdNZOdcdOdPZPdddQdRZQdSdTZRdUdVZSdWdXZTdS)ea Defines a minimal set of data types that allow to represent datasets with arbitrary nested structure, including objects of variable length, such as maps and lists. This defines a columnar storage format for such datasets on top of caffe2 tensors. In terms of capacity of representation, it can represent most of the data types supported by Parquet, ORC, DWRF file formats. See comments in operator_test/dataset_ops_test.py for an example and walkthrough on how to use schema to store and iterate through a structured in-memory dataset. N)core) workspace) BlobReference) OrderedDict namedtuple) basestring) viewitemsviewkeys viewvalues)islice)StringIO)Sequence:cCs.|r|rd|t|S|r|S|r&|SdSdS)Nz{}{}{})formatFIELD_SEPARATOR)prefixsuffixr8/tmp/pip-unpacked-wheel-ua33x9lu/caffe2/python/schema.py_join_field_name&srTcCs@t|tr|j|dSt|ttjfkr2t|dSt|dSdS)z:Clones/normalizes a field before adding it to a container. keep_blobsdtype)blobN) isinstanceFieldclonetypenprScalar)Zfield_or_type_or_blobrrrr_normalize_field1s    r" FeatureSpecZ feature_typeZ feature_namesZ feature_idsZfeature_is_request_onlyZdesired_hash_sizeZfeature_to_index)NNNNNNc@s"eZdZUdZdZeeed<dS)MetadataaSRepresents additional information associated with a scalar in schema. `categorical_limit` - for fields of integral type that are guaranteed to be non-negative it specifies the maximum possible value plus one. It's often used as a size of an embedding table. `expected_value` - anticipated average value of elements in the field. Usually makes sense for length fields of lists. `feature_specs` - information about the features that contained in this field. For example if field have more than 1 feature it can have list of feature names contained in this field.r __slots__N)__name__ __module__ __qualname____doc__r%r str__annotations__rrrrr$Ks  r$categorical_limitZexpected_valueZ feature_specs)NNNc@seZdZUdZdZeeed<ddZddZ dd Z d d Z d d Z ddZ ddZddZd$ddZddZddZd%ddZddZd d!Zd"d#ZdS)&rz4Represents an abstract field type in a dataset. )_parent_field_offsetsr%cCsFd|_d}g|_|D] }|j||t|7}q|j|dS)z:Derived classes must call this after their initialization.NrrN)r-r.appendlen field_names)selfchildrenoffsetchildrrr__init__is zField.__init__cCs |jddS)NFr)rr3rrr clone_schemasszField.clone_schemacCs tddS)z/Return the children field names for this field.Field is an abstract class.NNotImplementedErrorr8rrrr2vszField.field_namescCs tddS)z7Return the numpy.dtype for each of the children fields.r:Nr;r8rrr field_typeszszField.field_typescCs tddS)z4Return the Metadata for each of the children fields.r:Nr;r8rrrfield_metadata~szField.field_metadatacCs tddS)zReturn the list of blobs with contents for this Field. Values can either be all numpy.ndarray or BlobReference. If any of the fields doesn't have a blob, throws. r:Nr;r8rrr field_blobsszField.field_blobscCs tddS)zyReturn the list of all Scalar instances in the Field. The order is the same as for field_names() or field_blobs()r:Nr;r8rrr all_scalarsszField.all_scalarscCs tddS)z4Return True if every scalar of this field has blobs.r:Nr;r8rrr has_blobsszField.has_blobsTcCs tddS)z)Clone this Field along with its children.r:Nr;r3rrrrrsz Field.clonecCs||f|_dSN)r-)r3parentZ relative_idrrr _set_parentszField._set_parentcCs|}t||t|S)a Returns a slice representing the range of field ids that belong to this field. This slice can be used to index a list of fields. E.g.: >>> s = Struct( >>> ('a', Scalar()), >>> ('b', Struct( >>> ('b1', Scalar()), >>> ('b2', Scalar()), >>> )), >>> ('c', Scalar()), >>> ) >>> field_data = ['da', 'db1', 'db2', 'dc'] >>> field_data[s.b.split()] ['db1', 'db2'] )_child_base_idslicer1r2)r3Zbase_idrrrrGsz Field.sliceNcCs6|j\}}|dkrdn|j|}|r2|||7}|S)z"Get the base id of the given childNr)r-r.rF)r3Z child_indexpiposrrrrFs  zField._child_base_idcCs0||ko.||ko.||kS)zEquivalance of two schemas)r2r=r>)r3otherrrr__eq__s z Field.__eq__cCs tddS)Nr:r;r3indent str_bufferrrr _pprint_implszField._pprint_implcCs&t}|d||}||Sr/)r rPgetvalueclose)r3rOcontentsrrr__repr__s  zField.__repr__)T)N)r&r'r(r)r%r r*r+r7r9r2r=r>r?r@rArrErGrFrLrPrTrrrrrcs"    rcseZdZUdZdZeeed<dfdd ZddZ d d Z d d Z d dZ ddZ ddZdddZddZddZddZZS)Lista(Represents a variable-length list. Values of a list can also be complex fields such as Lists and Structs. In addition to the fields exposed by its `values` field, a List exposes an additional `lengths` field, which will contain the size of each list under the parent domain. )lengths_itemsr%Ncstt|tr$t|tstt||_nttj||_t||_|j |d|j |dt t | |j|jgdS)Nr) rrr!AssertionErrorr"rVr int32rWrEsuperrUr7)r3values lengths_blob __class__rrr7s   z List.__init__cCs|j}dgdd|DS)NrVcSsg|]}td|qSr\r.0vrrr sz$List.field_names..rWr2r3Z value_fieldsrrrr2s zList.field_namescCs|j|jSrC)rVr=rWr8rrrr=szList.field_typescCs|j|jSrC)rVr>rWr8rrrr>szList.field_metadatacCs|j|jSrC)rVr?rWr8rrrr?szList.field_blobscCs|j|jSrC)rVr@rWr8rrrr@szList.all_scalarscCs|jo|jSrC)rVrArWr8rrrrAszList.has_blobsTcCs"t|t|j|dt|j|dSNr)rr"rWrVrBrrrrs  z List.clonecCs||d|d|d|dd|jj|d|d|d|dd|jj|d|d|d|ddS) N zList( rX lengths= rNrO_items= ) )writerVrPrWrMrrrrPs zList._pprint_implcCsP|drt|t|jtr*t|j|S|dks:|dkr@|jStd|dS)\If the value of this list is a struct, allow to introspect directly into its fields.__valueitemsField not found in list: %s.N) startswithAttributeErrorrrWStructgetattrr3itemrrr __getattr__s   zList.__getattr__cCs`|td}t|dkr6|dkr&|jS|dkrP|jSn|ddkrP|j|dStd|dS)NrXrVr\rrt)splitrr1rVrWKeyErrorr3rznamesrrr __getitem__s   zList.__getitem__)N)Tr&r'r(r)r%r r*r+r7r2r=r>r?r@rArrPr{r __classcell__rrr^rrUs    rUcseZdZUdZdZeeed<dfdd ZddZ d d Z d d Z d dZ ddZ ddZdddZddZddZddZZS)ListWithEvictedzi This class is similar with List, but containing extra field evicted_values for LRU Hashing. )_evicted_valuesr%NcsJt|tr$t|tstt||_nttj||_tt |j ||ddS)Nr]) rrr!rYr"rr int64r[rr7)r3r\r]evicted_valuesr^rrr7%s   zListWithEvicted.__init__cCs$|j}dgdd|DdgS)NrVcSsg|]}td|qSr`rarbrrrre0sz/ListWithEvicted.field_names..rrfrgrrrr2-s zListWithEvicted.field_namescCs|j|j|jSrC)rVr=rWrr8rrrr=3szListWithEvicted.field_typescCs|j|j|jSrC)rVr>rWrr8rrrr>6szListWithEvicted.field_metadatacCs|j|j|jSrC)rVr?rWrr8rrrr?9szListWithEvicted.field_blobscCs|j|j|jSrC)rVr@rWrr8rrrr@<szListWithEvicted.all_scalarscCs|jo|j|jSrC)rVrArWrr8rrrrA?szListWithEvicted.has_blobsTcCs.t|t|j|dt|j|dt|j|dSrh)rr"rWrVrrBrrrrBs    zListWithEvicted.clonecCs|d|d|d|dd|jj|d|d|d|dd|jj|d|d|d|dd|jj|d|d|d|d dS) NrizListWithEvicted( rXrjrkrlrmz_evicted_values= rn)rorVrPrWrrMrrrrPIszListWithEvicted._pprint_implcCs^|drt||dkr |jSt|jtr8t|j|S|dksH|dkrN|jStd|dS)rprqrrrrsrtN)rurvrrrWrwrxryrrrr{Ts   zListWithEvicted.__getattr__cCsn|td}t|dkrD|dkr&|jS|dkr4|jS|dkr^|jSn|ddkr^|j|dStd|dS)NrXrVr\rrrt)r|rr1rVrWrr}r~rrrrbs   zListWithEvicted.__getitem__)NN)Trrrr^rrs   rcseZdZUdZdZeeed<fddZddZ dd Z d d Z d d Z ddZ ddZddZddZd-ddZddZddZddZdd Zd!d"Zd#d$Zfd%d&Zfd'd(Zd)d*Zd+d,ZZS).rwz?Represents a named list of fields sharing the same domain. )fields_frozenr%cs|D]8}t|dkst|ds(td|ddkstdqdd|D}t|_|D]p\}}t|krx|||\}}||jkr||j|<qXt|trt|j|tstd||j|||j|<qXt t |jD]\}\}}| ||qt t| t|jd |_d S) a  fields is a list of tuples in format of (name, field). The name is a string of nested name, e.g., `a`, `a:b`, `a:b:c`. For example Struct( ('a', Scalar()), ('b:c', Scalar()), ('b:d:e', Scalar()), ('b', Struct( ('f', Scalar()), )), ) is equal to Struct( ('a', Scalar()), ('b', Struct( ('c', Scalar()), ('d', Struct(('e', Scalar()))), ('f', Scalar()), )), ) rkrzField names cannot be emptyrVz.Struct cannot contain a field named `lengths`.cSsg|]\}}|t|fqSrr")rcnamefieldrrrresz#Struct.__init__..zDuplicate field name: %sTN)r1rYrrr_struct_from_nested_namerrw ValueError enumeraterrEr[r7r r)r3rrrid_r^rrr7xs0    zStruct.__init__cs>fdd|td}t|dks(t|d|d|fS)Ncs<|td}t|dkr|}n|d|}t|d|fSNrXr)r|rr1rw) nested_namerrZ added_fieldcreate_internalrrrs   z8Struct._struct_from_nested_name..create_internalrXrkr)r|rr1rY)r3rrrrrrrs  zStruct._struct_from_nested_namecCstt|jSrC)listrrr8rrr get_childrenszStruct.get_childrencs6g}t|jD]"\}|fdd|D7}q|S)Ncsg|]}t|qSrrarcfrrrresz&Struct.field_names..)rrr2)r3rrrrrr2szStruct.field_namescCs(g}t|jD]\}}||7}q|SrC)rrr=)r3typesrrrrrr=szStruct.field_typescCs(g}t|jD]\}}||7}q|SrC)rrr>)r3metadatarrrrrr>szStruct.field_metadatacCs(g}t|jD]\}}||7}q|SrC)rrr?)r3blobsrrrrrr?szStruct.field_blobscCs(g}t|jD]\}}||7}q|SrC)rrr@)r3scalarsrrrrrr@szStruct.all_scalarscCstddt|jDS)Ncss|]}|VqdSrC)rA)rcrrrr sz#Struct.has_blobs..)allr rr8rrrrAszStruct.has_blobsTcs$fddt|jD}t||S)Ncs g|]\}}|t|dfqS)rrrckrdrrrresz Struct.clone..)rrr)r3rZnormalized_fieldsrrrrs z Struct.clonec Csh|td}|j|dd}|dkr*dSt|dkr:|Sz||dWSttfk rbYdSXdSr)r|rrgetr1r} TypeError)r3rrrrrr_get_field_by_nested_names  z Struct._get_field_by_nested_namecCsn|d|dt|jD]:\}}|d|dd|d|j|d|dq|d|ddS) Nriz Struct( rXz{}= rkrlz) )rorrrrP)r3rNrOrrrrrrPs  zStruct._pprint_implcCs||}|dk SrC)rr3rzrrrr __contains__s zStruct.__contains__cCs t|jSrC)r1rr8rrr__len__szStruct.__len__cst|tst|tr:ttjtfdd|DSt|trZttt j|dS |}|dkrxt d||SdS)z item can be a tuple or list of ints or strings, or a single int or string. String item is a nested field name, e.g., "a", "a:b", "a:b:c". Int item is the index of a field at the first level of the Struct. cs*g|]"}t|tr|n||fqSr)rint)rcrkeysr3rrres  z&Struct.__getitem__..Nzfield "%s" not found) rrtupler rrwrnextr r rr}rrrrrs    zStruct.__getitem__cCs t|||S)z similar to python's dictionary get method, return field of item if found (i.e. self.item is valid) or otherwise return default_value it's a syntax suger of python's builtin getattr method )rx)r3rz default_valuerrrrsz Struct.getcsJ|drt|ztt|d|WStk rDt|YnXdS)Nrqr)rurvr[rw__getattribute__r}ryr^rrr{ s  zStruct.__getattr__cs4t|ddr|dstdtt|||dS)Nrrz1Struct.__setattr__() is disabled after __init__())rxrurr[rw __setattr__)r3keyrrr^rrr(szStruct.__setattr__cCst|tstSt|}|D]n\}}||kr<|||<q"||}t|trXt|tstdtt|dtt|d|||||<q"tt|S)a Allows to merge fields of two schema.Struct using '+' operator. If two Struct have common field names, the merge is conducted recursively. Here are examples: Example 1 s1 = Struct(('a', Scalar())) s2 = Struct(('b', Scalar())) s1 + s2 == Struct( ('a', Scalar()), ('b', Scalar()), ) Example 2 s1 = Struct( ('a', Scalar()), ('b', Struct(('c', Scalar()))), ) s2 = Struct(('b', Struct(('d', Scalar())))) s1 + s2 == Struct( ('a', Scalar()), ('b', Struct( ('c', Scalar()), ('d', Scalar()), )), ) Type of left_field, z, and type of right_field, z6, must both the Struct to allow merging of the field, ) rrwNotImplementedrrrr*rr)r3rKr4r right_field left_fieldrrr__add__1s,   zStruct.__add__cCst|tstSt|}|D]\}}||kr"||}t|t|krzt|trn||}|rn|||<q"||q"tdtt|dtt|d|q"t| S)a Allows to remove common fields of two schema.Struct from self by using '-' operator. If two Struct have common field names, the removal is conducted recursively. If a child struct has no fields inside, it will be removed from its parent. Here are examples: Example 1 s1 = Struct( ('a', Scalar()), ('b', Scalar()), ) s2 = Struct(('a', Scalar())) s1 - s2 == Struct(('b', Scalar())) Example 2 s1 = Struct( ('b', Struct( ('c', Scalar()), ('d', Scalar()), )) ) s2 = Struct( ('b', Struct(('c', Scalar()))), ) s1 - s2 == Struct( ('b', Struct( ('d', Scalar()), )), ) Example 3 s1 = Struct( ('a', Scalar()), ('b', Struct( ('d', Scalar()), )) ) s2 = Struct( ('b', Struct( ('c', Scalar()) ('d', Scalar()) )), ) s1 - s2 == Struct( ('a', Scalar()), ) rz*, is not the same as that of right_field, z%, yet they have the same field name, ) rrwrrrrpoprr*rs)r3rKr4rrrr6rrr__sub__`s20     zStruct.__sub__)T)r&r'r(r)r%r r*r+r7rrr2r=r>r?r@rArrrPrrrrr{rrrrrrr^rrwrs,  1    /rwcseZdZUdZdZeeed<d-fdd ZddZ d d Z d d Z d dZ ddZ ddZddZd.ddZddZddZeddZddZd d!Zd/d#d$Zd0d%d&Zd'd(Zd)d*Zd+d,ZZS)1r!a>Represents a typed scalar or tensor of fixed shape. A Scalar is a leaf in a schema tree, translating to exactly one tensor in the dataset's underlying storage. Usually, the tensor storing the actual values of this field is a 1D tensor, representing a series of values in its domain. It is possible however to have higher rank values stored as a Scalar, as long as all entries have the same shape. E.g.: Scalar(np.float64) Scalar field of type float64. Caffe2 will expect readers and datasets to expose it as a 1D tensor of doubles (vector), where the size of the vector is determined by this fields' domain. Scalar((np.int32, 5)) Tensor field of type int32. Caffe2 will expect readers and datasets to implement it as a 2D tensor (matrix) of shape (L, 5), where L is determined by this fields' domain. Scalar((str, (10, 20))) Tensor field of type str. Caffe2 will expect readers and datasets to implement it as a 3D tensor of shape (L, 10, 20), where L is determined by this fields' domain. If the field type is unknown at construction time, call Scalar(), that will default to np.void as its dtype. It is an error to pass a structured dtype to Scalar, since it would contain more than one field. Instead, use from_dtype, which will construct a nested `Struct` field reflecting the given dtype's structure. A Scalar can also contain a blob, which represents the value of this Scalar. A blob can be either a numpy.ndarray, in which case it contain the actual contents of the Scalar, or a BlobReference, which represents a blob living in a caffe2 Workspace. If blob of different types are passed, a conversion to numpy.ndarray is attempted. ) _metadatar_original_dtype_blobr%Ncs,d|_|j|||ddtt|gdS)NTunsafe)rsetr[r!r7)r3rrrr^rrr7szScalar.__init__cCsdgS)Nrrr8rrrr2szScalar.field_namescCs|jSrCrr8rrr field_typeszScalar.field_typecCs|jgSrCrr8rrrr=szScalar.field_typescCs|jgSrCrr8rrrr>szScalar.field_metadatacCs |jdk SrC)rr8rrrrAszScalar.has_blobscCs|jdk std|jgS)N Value is not set for this field.rrYr8rrrr?szScalar.field_blobscCs|gSrCrr8rrrr@szScalar.all_scalarsTcCst|j|r|jnd|jdS)Nrrr)r!rrrrBrrrrs  z Scalar.clonecCs|jdk std|jS)z+Gets the current blob of this Scalar field.Nrrr8rrrrsz Scalar.getcCs|S)zShortcut for self.get())rr8rrr__call__szScalar.__call__cCs|jSrCrr8rrrrszScalar.metadatacCs.t|tstdt|||_|dS)Nz!metadata must be Metadata, got {})rr$rYrrr_validate_metadata)r3rrrrr set_metadatas   zScalar.set_metadatacCsL|jdkrdS|jjdk rH|jdk rHt|jtjsHtdd|jdS)Nz6`categorical_limit` can be specified only in integral zfields but got {})rr,rr Z issubdtypeintegerrYrr8rrrr s   zScalar._validate_metadataFcCsj|jjtjkrT|rTt|tjs,td||jj|jjksTtd|jj|jj|j|j ||ddS)zrAr?r@rrrpropertyrrrrrrrPrrrrr^rr!s, ,    Sr!rr\cCstt||f||f|dS)zA map is a List of Struct containing keys and values fields. Optionally, you can provide custom name for the key and value fields. r)rUrw)rr\ keys_name values_namer]rrrMaps rcCstt||f||f||dS)z*A map with extra field evicted_values )r]r)rrw)rr\rrr]rrrrMapWithEvicteds rcstfddt|DS)Ncs g|]\}}d|f|fqS)z%s_%dr)rcrIr name_prefixrrreszNamedTuple..)rwr)rrrrr NamedTuples rcGs td|S)zP Creates a Struct with default, sequential, field names of given types. r)r)r)rrrrTuplesrrcCs0t|tst|dkstt|ftjg|S)z9 Creates a tuple of `num_field` untyped scalars. r)rrrYrr r) num_fieldsrrrrRawTuples rrcCst|tjs |}t||f}n$||j}||jkrDt|j|f}|jsRt|Sg}|jD]0\}\}}|dksxtd||t||df7}q\t |S)aSConstructs a Caffe2 schema from the given numpy's dtype. Numpy supports scalar, array-like and structured datatypes, as long as all the shapes are fixed. This function breaks down the given dtype into a Caffe2 schema containing `Struct` and `Scalar` types. Fields containing byte offsets are not currently supported. rz+Fields with byte offsets are not supported.) _outer_shape) rr rrrrr!rY from_dtyperw)rrr struct_fieldsrZfdtyper5rrrrs   rc@sFeZdZUdZdZeeed<dddZdddZ d d Z d d Z d S) _SchemaNodez7This is a private class used to represent a Schema Noderr4type_strrr%rcCs||_g|_||_d|_dSrCr)r3rrrrrr7sz_SchemaNode.__init__cCsB|jD] }|j|kr|j|kr|Sqt||}|j||SrC)r4rrrr0)r3rrr6rrr add_childs     z_SchemaNode.add_childc CsXddg}dddg}t|jdks*|jdk r@|jdkr:tS|jSg}|jD]}||jqJt|t|kr|jD] }|jdkr|}qr|}qrt||d|_d|_ |jSt|t|kr|jD]4}|jdkr|}q|jdkr|}q|}qt |||d|_d|_ |jSg}|jD]}||j|fq"t||_d|_ |jSdS) NrVr\rrrrUrrw) r1r4rrwr0rr get_fieldrUrr) r3Z list_namesZ map_namesZ child_namesr6Z values_fieldZ lengths_fieldZ key_fieldrrrrrsP               z_SchemaNode.get_fieldcCs:|jD] }|qtdt|jt|jdS)NzPrinting node: Name and type)r4print_recursivelyrinforr)r3r6rrrrs     z_SchemaNode.print_recursivelyN)r)r) r&r'r(r)r%r r*r+r7rrrrrrrrs   4rcCs*|dkrdgt|}|dkr,dgt|}|dkrBdgt|}t|t|ksZtdt|t|ksrtdt|t|kstdtdd}t||||D]~\}}}}|t}|} tt|D]V} || } d} d} | t|dkr|} t|||d } | | | }| dk r| |_ |} qq| S) zR Given a list of names, types, and optionally values, construct a Schema. Nz2col_names and col_types must have the same length.z5col_names and col_metadata must have the same length.z2col_names and col_blobs must have the same length.rootrwrrXr) r1rYrzipr|rranger!rrr)Z col_namesZ col_typesZ col_blobsZ col_metadatarZcol_nameZcol_typeZcol_blobcolumnscurrentrIrrrrrrrfrom_column_list&sP    rFcCst|tstdt|tr"|g}|}|}t|t|ksZtdt|t|ft||D]\}}|j||ddqd|S)zd Create a schema that clones the given schema, but containing the given list of values. z"Argument `schema` must be a Field.z%Values must have %d elements, got %d.Tr) rrrYrr9r@r1rr)schemar\rrecordrZscalarrrrrrfrom_blob_listUs rcCst|tr|St|ts"t|tr^tdd|D}|rJtdd|DStdd|DSn(t|tr~tddt|DSt |SdS)Ncss0|](}|tko&t|dko&t|dtVqdS)rkrN)rr1rrrrrrrkszas_record..cSsg|]\}}|t|fqSr as_recordrrrrrepszas_record..cSsg|] }t|qSrrrrrrrerscSsg|]\}}|t|fqSrrrrrrrets) rrrrrrwrdictrr")rrZ is_field_listrrrrgs  rcsVfddt|tst|}tdd|Ds8tfdd|D}t|||S)z Given a record containing BlobReferences, return a new record with same schema, containing numpy arrays, fetched from the current active workspace. cs,dkrtt|Sjt|SdSrC)rZ FetchBlobr*rfetch)rdwsrrrszFetchRecord..fetchcss|]}t|tVqdSrCrrrbrrrrszFetchRecord..csg|] }|qSrr)rcrr)rrrreszFetchRecord..)rrrYr?rr) blob_recordrrr?Z field_arraysr)rrr FetchRecordys  rcsfdd}t|tst|}tdd|Ds8tt|trJ|}t|t|ksjtdt|t||D]\}}|||qtdS)z Given a Record containing blob_references and arrays, which is either a list of numpy arrays or a Record containing numpy arrays, feeds the record to the current workspace. cs@dkrtt||n"t|jt||dSrC)rZFeedBlobr*Z create_blobrfeed)brdrrrrszFeedRecord..feedcss|]}t|tVqdSrCrrbrrrrszFeedRecord..z(Values must contain exactly %d ndarrays.N)rrrYr?rr1r)rZarraysrrr?rrrrr FeedRecords   rcs\t|tr*|}|jddd|St|ts.) rr!rrrrrYr2r)rrresultZ blob_refsrrr NewRecords  rcCs6t||}t||D]\}}|||q|S)z^ Given a record of arrays, returns a record of blobs, initialized with net.Const. )rrr?ZConst)rZ array_recordrrrrrr ConstRecords  rc Cs|st||}n|}t||D]t\}}z.t|}dgt|j}|jg|||dWq*t k rt d ||r|jg|dgdYq*Xq*|S)Nr)rrzBlob {} has type error)r) rArrr=r?data_type_for_dtyperrZ ConstantFillrrrr)rZschema_or_recordZ enforce_typesrZ blob_typerZ data_typerrrrInitEmptyRecords rcCst|t|SrC)rr2issubset)roriginal_schemarrris_schema_subsets  rcCsht|tstt|tst|r4||kr4dS|rL||krLdS|rd||krddSdS)NFT)rrrYr2r=r>)rrZcheck_field_namesZcheck_field_typesZcheck_field_metasrrr equal_schemassr cCs"t|}|dk rt||st|SrC)rr rY)rpreviousrrrr schema_checksr cCs6tD]\}}|j|kr|Sqtdt|jdS)NzUnknown dtype: )_DATA_TYPE_FOR_DTYPErrr*)rnp_typedtrrrrs   rcCs2tD]\}}||kr|Sqtdt|dS)NzUnknown core type: )r rr*)Z core_typer rrrrdtype_for_core_types  rcCs|D]}||qdSrC)r@r)rrrrrrattach_metadata_to_scalars%s r)T)rr\N)rr\NN)r)r)NNN)F)NF)N)F)TTF)N)Ur)loggingZnumpyr Z caffe2.pythonrrZcaffe2.python.corer collectionsrrZ past.builtinsrZ future.utilsrr r itertoolsr sixr typingr getLoggerr&rrrr"r#__new__ __defaults__r$objectrrUrrwr!rrrrrrrrrrrrrrrr*ZDataTypeSTRINGZfloat16ZFLOAT16Zfloat32FLOATZfloat64ZDOUBLEboolZBOOLZint8ZINT8Zint16ZINT16rZZINT32rZINT64Zuint8ZUINT8Zuint16ZUINT16r rr r rrrrrrrs           gSU7b   P /