U 3 d#@s|ddlZddlZddlZddlZddlmZddlmZddlm Z ddl m Z ddl mZddl mZGdd d e Z dS) N)StringIO)BotoClientError)Key)KeyFile) compute_hash)get_utf8_valuec seZdZdZd;fdd ZddZddZdddZ d?ddZ d@ddZ dAddZdBddZddZd d!ZdCd"d#Zd$d%ZdDd'd(ZdEd)d*ZdFd+d,Zfd-d.ZdGd/d0ZdHd1d2ZdId3d4ZdJd5d6ZdKd7d8ZdLd9d:ZZS)Mra Represents a key (object) in a GS bucket. :ivar bucket: The parent :class:`boto.gs.bucket.Bucket`. :ivar name: The name of this Key object. :ivar metadata: A dictionary containing user metadata that you wish to store with the object or that has been retrieved from an existing object. :ivar cache_control: The value of the `Cache-Control` HTTP header. :ivar content_type: The value of the `Content-Type` HTTP header. :ivar content_encoding: The value of the `Content-Encoding` HTTP header. :ivar content_disposition: The value of the `Content-Disposition` HTTP header. :ivar content_language: The value of the `Content-Language` HTTP header. :ivar etag: The `etag` associated with this object. :ivar last_modified: The string timestamp representing the last time this object was modified in GS. :ivar owner: The ID of the owner of this object. :ivar storage_class: The storage class of the object. Currently, one of: STANDARD | DURABLE_REDUCED_AVAILABILITY. :ivar md5: The MD5 hash of the contents of the object. :ivar size: The size, in bytes, of the object. :ivar generation: The generation number of the object. :ivar metageneration: The generation number of the object metadata. :ivar encrypted: Whether the object is encrypted while at rest on the server. :ivar cloud_hashes: Dictionary of checksums as supplied by the storage provider. Ncs0tt|j||d||_d|_i|_d|_dS)N)bucketname)superr__init__ generationZmeta_generation cloud_hashescomponent_count)selfrr r  __class__//tmp/pip-unpacked-wheel-d7dsrkjd/boto/gs/key.pyr As z Key.__init__cCsN|jr|jrd|j|jf}nd}|jrz)r metagenerationrr )rZver_strrrr__repr__Hs  z Key.__repr__cCs|dkr||_n|dkr ||_n|dkr@|dkr8d|_qd|_nz|dkrP||_nj|dkrdt||_nV|d krt||_nF|d kr~n<|d kr||_n,|d kr||_n|d kr||_ n t |||dS)NrETagZIsLatesttrueTFZ LastModifiedSizeZ StorageClassZOwnerZ VersionIdZ GenerationZMetaGeneration) r etagZ is_latestZ last_modifiedintsize storage_class version_idr rsetattr)rr value connectionrrr endElementRs. zKey.endElementFcCs |dd|_|dd|_dS)Nzx-goog-metagenerationx-goog-generation) getheaderrr )rrespforcerrrhandle_version_headersmszKey.handle_version_headerscCsdS)Nr)rresponserrrhandle_restore_headersqszKey.handle_restore_headerscCs|D]\}}|dkrJ|dD](}|dd\}}t||j|<qq|dkr^t||_q|dkrn||_q|dkr~||_q|dkrt||_ q|d kr||_ qdS) Nz x-goog-hash,=zx-goog-component-countr#zx-goog-stored-content-encodingzx-goog-stored-content-lengthzx-goog-storage-class) splitstripbinascii a2b_base64r rrr content_encodingrr)rheaderskeyr Z hash_pairZalg b64_digestrrrhandle_addl_headersts   zKey.handle_addl_headersrcs<|jr |r|d7}|d|j7}tt|j||||ddS)a Open this key for reading :type headers: dict :param headers: Headers to pass in the web request :type query_args: string :param query_args: Arguments to pass in the query string (ie, 'torrent') :type override_num_retries: int :param override_num_retries: If not None will override configured num_retries parameter for underlying GET. :type response_headers: dict :param response_headers: A dictionary containing HTTP headers/values that will override any headers associated with the stored object in the response. See http://goo.gl/EWOPb for details. & generation=%s)r2 query_argsoverride_num_retriesresponse_headersN)r r r open_read)rr2r8r9r:rrrr;sz Key.open_read c Cs4d} |jrd|jg} |j||||||| | ddS)Nr7)r2cbnum_cbr9r: hash_algsr8)r Z_get_file_internal) rfpr2r=r>torrentrr9r:r?r8rrrget_files  z Key.get_filec CsH|jdk rD|r*|j|||||||| dn|j|||||||| ddS)aG Retrieve an object from GCS using the name of the Key object as the key in GCS. Write the contents of the object to the file pointed to by 'fp'. :type fp: File -like object :param fp: :type headers: dict :param headers: additional HTTP headers that will be sent with the GET request. :type cb: function :param cb: a callback function that will be called to report progress on the upload. The callback should accept two integer parameters, the first representing the number of bytes that have been successfully transmitted to GCS and the second representing the size of the to be transmitted object. :type cb: int :param num_cb: (optional) If a callback is specified with the cb parameter this parameter determines the granularity of the callback by defining the maximum number of times the callback will be called during the file transfer. :type torrent: bool :param torrent: If True, returns the contents of a torrent file as a string. :type res_upload_handler: ResumableDownloadHandler :param res_download_handler: If provided, this handler will perform the download. :type response_headers: dict :param response_headers: A dictionary containing HTTP headers/values that will override any headers associated with the stored object in the response. See http://goo.gl/sMkcC for details. N)rArr?)rArr:r?)rrB) rr@r2r=r>rArZres_download_handlerr:r?rrrget_contents_to_files/ zKey.get_contents_to_filecCs"t|||d\}}}||_||fS)at :type fp: file :param fp: File pointer to the file to hash. The file pointer will be reset to the same position before the method returns. :type algorithm: zero-argument constructor for hash objects that implements update() and digest() (e.g. hashlib.md5) :type size: int :param size: (optional) The Maximum number of bytes to read from the file pointer (fp). This is useful when uploading a file in multiple parts where the file is being split in place into different parts. Less bytes may be available. )rZhash_algorithm)rr)rr@ algorithmr hex_digestr4 data_sizerrrrs zKey.compute_hashc Cs|j||||||||ddS)a; Upload a file to GCS. :type fp: file :param fp: The file pointer to upload. The file pointer must point at the offset from which you wish to upload. ie. if uploading the full file, it should point at the start of the file. Normally when a file is opened for reading, the fp will point at the first byte. See the bytes parameter below for more info. :type headers: dict :param headers: The headers to pass along with the PUT request :type num_cb: int :param num_cb: (optional) If a callback is specified with the cb parameter this parameter determines the granularity of the callback by defining the maximum number of times the callback will be called during the file transfer. Providing a negative integer will cause your callback to be called with each buffer read. :type query_args: string :param query_args: Arguments to pass in the query string. :type chunked_transfer: boolean :param chunked_transfer: (optional) If true, we use chunked Transfer-Encoding. :type size: int :param size: (optional) The Maximum number of bytes to read from the file pointer (fp). This is useful when uploading a file in multiple parts where you are splitting the file up into different ranges to be uploaded. If not specified, the default behaviour is to read all bytes from the file pointer. Less bytes may be available. :type hash_algs: dictionary :param hash_algs: (optional) Dictionary of hash algorithms and corresponding hashing class that implements update() and digest(). Defaults to {'md5': hashlib.md5}. )r2r=r>r8chunked_transferrr?N)Z_send_file_internal) rr@r2r=r>r8rGrr?rrr send_file s - z Key.send_filecCs|jj|j|j|j|dS)N)rr r2)rZ delete_keyr rr )rr2rrrdelete=sz Key.deletecCs"|}|||||dS)a Convenience method that provides a quick way to add an email grant to a key. This method retrieves the current ACL, creates a new grant based on the parameters passed in, adds that grant to the ACL and then PUT's the new ACL back to GS. :type permission: string :param permission: The permission being granted. Should be one of: READ|FULL_CONTROL See http://code.google.com/apis/storage/docs/developer-guide.html#authorization for more details on permissions. :type email_address: string :param email_address: The email address associated with the Google account to which you are granting the permission. N)get_acladd_email_grantset_acl)r permission email_addressaclrrrrKBs zKey.add_email_grantcCs"|}|||||dS)a Convenience method that provides a quick way to add a canonical user grant to a key. This method retrieves the current ACL, creates a new grant based on the parameters passed in, adds that grant to the ACL and then PUT's the new ACL back to GS. :type permission: string :param permission: The permission being granted. Should be one of: READ|FULL_CONTROL See http://code.google.com/apis/storage/docs/developer-guide.html#authorization for more details on permissions. :type user_id: string :param user_id: The canonical user id associated with the GS account to which you are granting the permission. N)rJadd_user_grantrL)rrMZuser_idrOrrrrPWs zKey.add_user_grantcCs*|j|d}||||j||ddS)a Convenience method that provides a quick way to add an email group grant to a key. This method retrieves the current ACL, creates a new grant based on the parameters passed in, adds that grant to the ACL and then PUT's the new ACL back to GS. :type permission: string :param permission: The permission being granted. Should be one of: READ|FULL_CONTROL See http://code.google.com/apis/storage/docs/developer-guide.html#authorization for more details on permissions. :type email_address: string :param email_address: The email address associated with the Google Group to which you are granting the permission. )r2N)rJadd_group_email_grantrL)rrMrNr2rOrrrrQls  zKey.add_group_email_grantcCs"|}|||||dS)a Convenience method that provides a quick way to add a canonical group grant to a key. This method retrieves the current ACL, creates a new grant based on the parameters passed in, adds that grant to the ACL and then PUT's the new ACL back to GS. :type permission: string :param permission: The permission being granted. Should be one of: READ|FULL_CONTROL See http://code.google.com/apis/storage/docs/developer-guide.html#authorization for more details on permissions. :type group_id: string :param group_id: The canonical group id associated with the Google Groups account you are granting the permission to. N)rJadd_group_grantrL)rrMZgroup_idrOrrrrRs zKey.add_group_grantTc Cs|jjj} |r| rtd|p i}|r0||| j<| rD|dtjnbt|t s| } |dtj | | kr|dtj| | kr|| t d|| t |dr|j|_|jdk rt|t r|}|j| } | |_td|jr|jd}|tt|f}| r*| |_n4| } |dtj | | |_|| |j} |dkrt||| }|d|_|d|_|jdkr|j|_|s|j|jrdS| dk rt| |d <|r||||||n|j||||| d dS) a Store an object in GS using the name of the Key object as the key in GS and the contents of the file pointed to by 'fp' as the contents. :type fp: file :param fp: The file whose contents are to be uploaded. :type headers: dict :param headers: (optional) Additional HTTP headers to be sent with the PUT request. :type replace: bool :param replace: (optional) If this parameter is False, the method will first check to see if an object exists in the bucket with the same key. If it does, it won't overwrite it. The default value is True which will overwrite the object. :type cb: function :param cb: (optional) Callback function that will be called to report progress on the upload. The callback should accept two integer parameters, the first representing the number of bytes that have been successfully transmitted to GS and the second representing the total number of bytes that need to be transmitted. :type num_cb: int :param num_cb: (optional) If a callback is specified with the cb parameter, this parameter determines the granularity of the callback by defining the maximum number of times the callback will be called during the file transfer. :type policy: :class:`boto.gs.acl.CannedACLStrings` :param policy: (optional) A canned ACL policy that will be applied to the new key in GS. :type md5: tuple :param md5: (optional) A tuple containing the hexdigest version of the MD5 checksum of the file as the first element and the Base64-encoded version of the plain checksum as the second element. This is the same format returned by the compute_md5 method. If you need to compute the MD5 for any reason prior to upload, it's silly to have to do it twice so this param, if present, will be used as the MD5 values of the file. Otherwise, the checksum will be computed. :type res_upload_handler: :py:class:`boto.gs.resumable_upload_handler.ResumableUploadHandler` :param res_upload_handler: (optional) If provided, this handler will perform the upload. :type size: int :param size: (optional) The Maximum number of bytes to read from the file pointer (fp). This is useful when uploading a file in multiple parts where you are splitting the file up into different ranges to be uploaded. If not specified, the default behaviour is to read all bytes from the file pointer. Less bytes may be available. Notes: 1. The "size" parameter currently cannot be used when a resumable upload handler is given but is still useful for uploading part of a file as implemented by the parent class. 2. At present Google Cloud Storage does not support multipart uploads. :type rewind: bool :param rewind: (optional) If True, the file pointer (fp) will be rewound to the start before any bytes are read from it. The default behaviour is False which reads from the current position of the file pointer (fp). :type if_generation: int :param if_generation: (optional) If set to a generation number, the object will only be written to if its current generation number is this value. If set to the value 0, the object will only be written if it doesn't already exist. :rtype: int :return: The number of bytes written to the key. TODO: At some point we should refactor the Bucket and Key classes, to move functionality common to all providers into a parent class, and provider-specific functionality into subclasses (rather than just overriding/sharing code the way it currently works). z1"size" param not supported for resumable uploads.rz8fp is at EOF. Use rewind option or seek() to data start.r Nz^"[a-fA-F0-9]{32}"$"r,x-goog-if-generation-match)r) rr!providerrZ acl_headerseekosSEEK_SET isinstancertellSEEK_ENDAttributeErrorhasattrr pathZgetkeyrrematchrr.base64 b64encoder/ unhexlifyZ compute_md5md5 base64md5lookupstrrH)rr@r2replacer=r>policyrdres_upload_handlerrrewind if_generationrUsposr3rrrrset_contents_from_filesdY                   zKey.set_contents_from_filec Cs<i|_t|d"} |j| ||||||| | d W5QRXdS)a Store an object in GS using the name of the Key object as the key in GS and the contents of the file named by 'filename'. See set_contents_from_file method for details about the parameters. :type filename: string :param filename: The name of the file that you want to put onto GS. :type headers: dict :param headers: (optional) Additional headers to pass along with the request to GS. :type replace: bool :param replace: (optional) If True, replaces the contents of the file if it already exists. :type cb: function :param cb: (optional) Callback function that will be called to report progress on the upload. The callback should accept two integer parameters, the first representing the number of bytes that have been successfully transmitted to GS and the second representing the total number of bytes that need to be transmitted. :type num_cb: int :param num_cb: (optional) If a callback is specified with the cb parameter this parameter determines the granularity of the callback by defining the maximum number of times the callback will be called during the file transfer. :type policy: :py:attribute:`boto.gs.acl.CannedACLStrings` :param policy: (optional) A canned ACL policy that will be applied to the new key in GS. :type md5: tuple :param md5: (optional) A tuple containing the hexdigest version of the MD5 checksum of the file as the first element and the Base64-encoded version of the plain checksum as the second element. This is the same format returned by the compute_md5 method. If you need to compute the MD5 for any reason prior to upload, it's silly to have to do it twice so this param, if present, will be used as the MD5 values of the file. Otherwise, the checksum will be computed. :type res_upload_handler: :py:class:`boto.gs.resumable_upload_handler.ResumableUploadHandler` :param res_upload_handler: (optional) If provided, this handler will perform the upload. :type if_generation: int :param if_generation: (optional) If set to a generation number, the object will only be written to if its current generation number is this value. If set to the value 0, the object will only be written if it doesn't already exist. rbrlN)Z local_hashesopenrn) rfilenamer2rhr=r>rirdZreduced_redundancyrjrlr@rrrset_contents_from_filenameHs> zKey.set_contents_from_filenamec Cs>d|_d|_tt|} |j| |||||||d} | | S)aA Store an object in GCS using the name of the Key object as the key in GCS and the string 's' as the contents. See set_contents_from_file method for details about the parameters. :type headers: dict :param headers: Additional headers to pass along with the request to AWS. :type replace: bool :param replace: If True, replaces the contents of the file if it already exists. :type cb: function :param cb: a callback function that will be called to report progress on the upload. The callback should accept two integer parameters, the first representing the number of bytes that have been successfully transmitted to GCS and the second representing the size of the to be transmitted object. :type cb: int :param num_cb: (optional) If a callback is specified with the cb parameter this parameter determines the granularity of the callback by defining the maximum number of times the callback will be called during the file transfer. :type policy: :class:`boto.gs.acl.CannedACLStrings` :param policy: A canned ACL policy that will be applied to the new key in GCS. :type md5: A tuple containing the hexdigest version of the MD5 checksum of the file as the first element and the Base64-encoded version of the plain checksum as the second element. This is the same format returned by the compute_md5 method. :param md5: If you need to compute the MD5 for any reason prior to upload, it's silly to have to do it twice so this param, if present, will be used as the MD5 values of the file. Otherwise, the checksum will be computed. :type if_generation: int :param if_generation: (optional) If set to a generation number, the object will only be written to if its current generation number is this value. If set to the value 0, the object will only be written if it doesn't already exist. Nrp)rdrerrrnclose) rsr2rhr=r>rirdrlr@rrrrset_contents_from_strings6 zKey.set_contents_from_stringcsJ|dd}|dk r4|di}t||d<||d<tt|j||dS)a1 Store an object using the name of the Key object as the key in cloud and the contents of the data stream pointed to by 'fp' as the contents. The stream object is not seekable and total size is not known. This has the implication that we can't specify the Content-Size and Content-MD5 in the header. So for huge uploads, the delay in calculating MD5 is avoided but with a penalty of inability to verify the integrity of the uploaded data. :type fp: file :param fp: the file whose contents are to be uploaded :type headers: dict :param headers: additional HTTP headers to be sent with the PUT request. :type replace: bool :param replace: If this parameter is False, the method will first check to see if an object exists in the bucket with the same key. If it does, it won't overwrite it. The default value is True which will overwrite the object. :type cb: function :param cb: a callback function that will be called to report progress on the upload. The callback should accept two integer parameters, the first representing the number of bytes that have been successfully transmitted to GS and the second representing the total number of bytes that need to be transmitted. :type num_cb: int :param num_cb: (optional) If a callback is specified with the cb parameter, this parameter determines the granularity of the callback by defining the maximum number of times the callback will be called during the file transfer. :type policy: :class:`boto.gs.acl.CannedACLStrings` :param policy: A canned ACL policy that will be applied to the new key in GS. :type size: int :param size: (optional) The Maximum number of bytes to read from the file pointer (fp). This is useful when uploading a file in multiple parts where you are splitting the file up into different ranges to be uploaded. If not specified, the default behaviour is to read all bytes from the file pointer. Less bytes may be available. :type if_generation: int :param if_generation: (optional) If set to a generation number, the object will only be written to if its current generation number is this value. If set to the value 0, the object will only be written if it doesn't already exist. rlNr2rT)popgetrgr rset_contents_from_stream)rargskwargsrlr2rrrrzs 9   zKey.set_contents_from_streamcCs(|jdk r$|jj||j||||ddS)aSets the ACL for this object. :type acl_or_str: string or :class:`boto.gs.acl.ACL` :param acl_or_str: A canned ACL string (see :data:`~.gs.acl.CannedACLStrings`) or an ACL object. :type headers: dict :param headers: Additional headers to set during the request. :type generation: int :param generation: If specified, sets the ACL for a specific generation of a versioned object. If not specified, the current version is modified. :type if_generation: int :param if_generation: (optional) If set to a generation number, the acl will only be updated if its current generation number is this value. :type if_metageneration: int :param if_metageneration: (optional) If set to a metageneration number, the acl will only be updated if its current metageneration number is this value. Nr2r rlif_metageneration)rrLr )rZ acl_or_strr2r rlr~rrrrL s  z Key.set_aclcCs"|jdk r|jj|j||dSdS)aQReturns the ACL of this object. :param dict headers: Additional headers to set during the request. :param int generation: If specified, gets the ACL for a specific generation of a versioned object. If not specified, the current version is returned. :rtype: :class:`.gs.acl.ACL` Nr2r )rrJr rr2r rrrrJ,s  z Key.get_aclcCs"|jdk r|jj|j||dSdS)aGReturns the ACL string of this object. :param dict headers: Additional headers to set during the request. :param int generation: If specified, gets the ACL for a specific generation of a versioned object. If not specified, the current version is returned. :rtype: str Nr)r get_xml_aclr rrrrr;s  zKey.get_xml_aclcCs(|jdk r$|jj||j||||dSdS)a`Sets this objects's ACL to an XML string. :type acl_str: string :param acl_str: A string containing the ACL XML. :type headers: dict :param headers: Additional headers to set during the request. :type generation: int :param generation: If specified, sets the ACL for a specific generation of a versioned object. If not specified, the current version is modified. :type if_generation: int :param if_generation: (optional) If set to a generation number, the acl will only be updated if its current generation number is this value. :type if_metageneration: int :param if_metageneration: (optional) If set to a metageneration number, the acl will only be updated if its current metageneration number is this value. Nr})r set_xml_aclr rZacl_strr2r rlr~rrrrJs  zKey.set_xml_aclcCs(|jdk r$|jj||j||||dSdS)aSets this objects's ACL using a predefined (canned) value. :type acl_str: string :param acl_str: A canned ACL string. See :data:`~.gs.acl.CannedACLStrings`. :type headers: dict :param headers: Additional headers to set during the request. :type generation: int :param generation: If specified, sets the ACL for a specific generation of a versioned object. If not specified, the current version is modified. :type if_generation: int :param if_generation: (optional) If set to a generation number, the acl will only be updated if its current generation number is this value. :type if_metageneration: int :param if_metageneration: (optional) If set to a metageneration number, the acl will only be updated if its current metageneration number is this value. Nr})rset_canned_aclr rrrrrhs zKey.set_canned_aclc Csg}|D]H}|jj|jjkr$tdd}|jr%sz(%s%sz#%sz Content-TypePUTcompose)r2r8datai+r#)rr rr rgappendjoinr!Z make_requestrstatusrUZstorage_response_errorreasonreadr$) r components content_typer2Z compose_reqr3Zgeneration_tagZcompose_req_xmlr%rrrrsF  z Key.compose)NNN)F)NrNN)NNr<FNNNN)NNr<FNNNN)N)NNr<NFNN)N)N) NTNr<NNNNFN) NTNr<NNNNN)NTNr<NNN)NNNN)NN)NN)NNNN)NNNN)NN)__name__ __module__ __qualname____doc__r rr"r'r)r5r;rBrCrrHrIrKrPrQrRrnrsrwrzrLrJrrrr __classcell__rrrrr"s     ;  2   3 E @ @     #r)rar/rWr_Z boto.compatrZboto.exceptionrZ boto.s3.keyrZS3KeyZboto.s3.keyfilerZ boto.utilsrrrrrrs