U O8c??@shddlZddlZddlmZeeZGdddZGdddeZGdddeZ Gd d d eZ dS) N) get_callbacksc@sveZdZdZdddZddZedd Zd d Zd d Z ddZ ddZ ddZ ddZ ddZddZddZdS)TaskzA task associated to a TransferFuture request This is a base class for other classes to subclass from. All subclassed classes must implement the main() method. NFcCsP||_||_|jdkri|_||_|dkr0i|_||_|jdkrFg|_||_dS)a :type transfer_coordinator: s3transfer.futures.TransferCoordinator :param transfer_coordinator: The context associated to the TransferFuture for which this Task is associated with. :type main_kwargs: dict :param main_kwargs: The keyword args that can be immediately supplied to the _main() method of the task :type pending_main_kwargs: dict :param pending_main_kwargs: The keyword args that are depended upon by the result from a dependent future(s). The result returned by the future(s) will be used as the value for the keyword argument when _main() is called. The values for each key can be: * a single future - Once completed, its value will be the result of that single future * a list of futures - Once all of the futures complete, the value used will be a list of each completed future result value in order of when they were originally supplied. :type done_callbacks: list of callbacks :param done_callbacks: A list of callbacks to call once the task is done completing. Each callback will be called with no arguments and will be called no matter if the task succeeds or an exception is raised. :type is_final: boolean :param is_final: True, to indicate that this task is the final task for the TransferFuture request. By setting this value to True, it will set the result of the entire TransferFuture to the result returned by this task's main() method. N)_transfer_coordinator _main_kwargs_pending_main_kwargs_done_callbacks _is_final)selfZtransfer_coordinatorZ main_kwargsZpending_main_kwargsZdone_callbacksis_finalr 4/tmp/pip-unpacked-wheel-6hpttf6a/s3transfer/tasks.py__init__s(  z Task.__init__cCs6dddddddg}||j|}d|jj|jj|S) NbucketkeyZ part_numberZfinal_filenametransfer_futureoffset extra_argsz{}(transfer_id={}, {}))"_get_kwargs_with_params_to_includerformat __class____name__r transfer_id)r Zparams_to_displayZmain_kwargs_to_displayr r r __repr__Ts" z Task.__repr__cCs|jjS)z8The id for the transfer request that the task belongs to)rr)r r r r riszTask.transfer_idcCs&i}|D]}||kr||||<q|SNr )r kwargsincludefiltered_kwargsparamr r r rns z'Task._get_kwargs_with_params_to_includecCs,i}|D]\}}||krq |||<q |Sr)items)r rexcluderrvaluer r r "_get_kwargs_with_params_to_excludeus  z'Task._get_kwargs_with_params_to_excludec Csz`z.||}|js.||WW6SWn,tk r\}z| |W5d}~XYnXW5|jD] }|qh|jr|jXdS)z9The callable to use when submitting a Task to an executorN) rrr announce_done_wait_on_dependent_futures_get_all_main_kwargsdone _execute_main Exception_log_and_set_exception)r Z done_callbackrer r r __call__}s   z Task.__call__cCsJdg}|||}td|d||jf|}|jrF|j||S)NdatazExecuting task z with kwargs )r!loggerdebug_mainrrZ set_result)r rZparams_to_excludeZkwargs_to_displayZ return_valuer r r r&s  zTask._execute_maincCstjddd|j|dS)NzException raised.T)exc_info)r,r-rZ set_exception)r exceptionr r r r(szTask._log_and_set_exceptioncKs tddS)zThe method that will be ran in the executor This method must be implemented by subclasses from Task. main() can be implemented with any arguments decided upon by the subclass. z_main() must be implementedNNotImplementedError)r rr r r r.sz Task._maincCsFg}|jD](\}}t|tr,||q||q||dSr)rr isinstancelistextendappend_wait_until_all_complete)r Zfutures_to_wait_on_futurer r r r#s    zTask._wait_on_dependent_futuresc CsXtd|||D]4}ztd|||Wqtk rDYqXqtd|dS)Nz-%s about to wait for the following futures %sz%s about to wait for %sz%%s done waiting for dependent futures)r,r-resultr')r Zfuturesr9r r r r7s  zTask._wait_until_all_completecCs\t|j}|jD]@\}}t|trFg}|D]}||q0n|}|||<q|Sr)copyrrrr3r4r6r:)r rrZ pending_valuer:r9r r r r$s   zTask._get_all_main_kwargs)NNNF)r __module__ __qualname____doc__r rpropertyrrr!r*r&r(r.r#r7r$r r r r rs$  8  rc@s(eZdZdZddZddZddZdS) SubmissionTaskzA base class for any submission task Submission tasks are the top-level task used to submit a series of tasks to execute a particular transfer. c KszF|jt|d}|D] }|q|j|jfd|i|Wn>tk r}z ||||jW5d}~XYnXdS)a> :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The transfer future associated with the transfer request that tasks are being submitted for :param kwargs: Any additional kwargs that you may want to pass to the _submit() method ZqueuedrN) rZset_status_to_queuedrZset_status_to_running_submit BaseExceptionr(+_wait_for_all_submitted_futures_to_completer")r rrZon_queued_callbacksZon_queued_callbackr)r r r r.s     zSubmissionTask._maincKs tddS)aXThe submission method to be implemented :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The transfer future associated with the transfer request that tasks are being submitted for :param kwargs: Any additional keyword arguments you want to be passed in z_submit() must be implementedNr1)r rrr r r rA&s zSubmissionTask._submitcCs2|jj}|r.|||jj}||kr(q.|}qdSr)rZassociated_futuresr7)r Zsubmitted_futuresZpossibly_more_submitted_futuresr r r rC2s  z:SubmissionTask._wait_for_all_submitted_futures_to_completeN)rr<r=r>r.rArCr r r r r@s/ r@c@seZdZdZddZdS)CreateMultipartUploadTaskz#Task to initiate a multipart uploadcCs8|jf||d|}|d}|jj|j|||d|S)aq :param client: The client to use when calling CreateMultipartUpload :param bucket: The name of the bucket to upload to :param key: The name of the key to upload to :param extra_args: A dictionary of any extra arguments that may be used in the initialization. :returns: The upload id of the multipart upload )BucketKeyUploadId)rErFrG)Zcreate_multipart_uploadrZadd_failure_cleanupZabort_multipart_upload)r clientrrrresponse upload_idr r r r.Qs zCreateMultipartUploadTask._mainNrr<r=r>r.r r r r rDNsrDc@seZdZdZddZdS)CompleteMultipartUploadTaskz#Task to complete a multipart uploadcCs"|jf|||d|id|dS)az :param client: The client to use when calling CompleteMultipartUpload :param bucket: The name of the bucket to upload to :param key: The name of the key to upload to :param upload_id: The id of the upload :param parts: A list of parts to use to complete the multipart upload:: [{'Etag': etag_value, 'PartNumber': part_number}, ...] Each element in the list consists of a return value from ``UploadPartTask.main()``. :param extra_args: A dictionary of any extra arguments that may be used in completing the multipart transfer. ZParts)rErFrGZMultipartUploadN)Zcomplete_multipart_upload)r rHrrrJpartsrr r r r.nsz!CompleteMultipartUploadTask._mainNrKr r r r rLksrL) r;loggingZs3transfer.utilsr getLoggerrr,rr@rDrLr r r r  s  \^