File: //lib/python2.7/site-packages/s3transfer/futures.pyc
�
P$�_c @ s� d d l m Z d d l m Z d d l Z d d l Z d d l Z d d l Z d d l m Z d d l m
Z
d d l m Z m
Z
d d l m Z d d l m Z e j e � Z d e f d
� � YZ d e f d � � YZ d
e f d � � YZ d e f d � � YZ d e f d � � YZ d e f d � � YZ d e f d � � YZ d e f d � � YZ d e f d � � YZ d e f d � � YZ e d d g � Z e d � Z e d � Z! d S(! i����( t futures( t
namedtupleN( t MAXINT( t six( t CancelledErrort TransferNotDoneError( t FunctionContainer( t
TaskSemaphoret BaseTransferFuturec B s2 e Z e d � � Z d � Z d � Z d � Z RS( c C s t d � � d S( s- The metadata associated to the TransferFuturet metaN( t NotImplementedError( t self( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR s c C s t d � � d S( sm Determines if a TransferFuture has completed
:returns: True if completed. False, otherwise.
s done()N( R
( R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyt done$ s c C s t d � � d S( s� Waits until TransferFuture is done and returns the result
If the TransferFuture succeeded, it will return the result. If the
TransferFuture failed, it will raise the exception associated to the
failure.
s result()N( R
( R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyt result+ s c C s t d � � d S( s6 Cancels the request associated with the TransferFutures cancel()N( R
( R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyt cancel4 s ( t __name__t
__module__t propertyR R R
R ( ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR s t BaseTransferMetac B s5 e Z e d � � Z e d � � Z e d � � Z RS( c C s t d � � d S( s* The call args used in the transfer requestt call_argsN( R
( R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR : s c C s t d � � d S( s The unique id of the transfert transfer_idN( R
( R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR ? s c C s t d � � d S( s. A dictionary that requesters can store data int user_contextN( R
( R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR D s ( R R R R R R ( ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR 9 s t TransferFuturec B sJ e Z d d d � Z e d � � Z d � Z d � Z d � Z d � Z RS( c C sL | | _ | d k r$ t � | _ n | | _ | d k rH t � | _ n d S( s� The future associated to a submitted transfer request
:type meta: TransferMeta
:param meta: The metadata associated to the request. This object
is visible to the requester.
:type coordinator: TransferCoordinator
:param coordinator: The coordinator associated to the request. This
object is not visible to the requester.
N( t _metat Nonet TransferMetat _coordinatort TransferCoordinator( R R t coordinator( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyt __init__K s c C s | j S( N( R ( R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR ^ s c C s
| j j � S( N( R R ( R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR b s c C s; y | j j � SWn# t k
r6 } | j � | � n Xd S( N( R R
t KeyboardInterruptR ( R t e( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR
e s
c C s | j j � d S( N( R R ( R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR o s c C s5 | j � s t d � � n | j j | d t �d S( s! Sets the exception on the future.s? set_exception can only be called once the transfer is complete.t overrideN( R R R t
set_exceptiont True( R t exception( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR! r s N(
R R R R R R R R
R R! ( ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR J s
R c B sb e Z d Z d d d � Z e d � � Z e d � � Z e d � � Z e d � � Z d � Z
RS( s' Holds metadata about the TransferFuturec C s( | | _ | | _ d | _ i | _ d S( N( t
_call_argst _transfer_idR t _sizet
_user_context( R R R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR } s c C s | j S( s* The call args used in the transfer request( R$ ( R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR � s c C s | j S( s The unique id of the transfer( R% ( R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR � s c C s | j S( s) The size of the transfer request if known( R& ( R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyt size� s c C s | j S( s. A dictionary that requesters can store data in( R' ( R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR � s c C s
| | _ d S( s� A method to provide the size of a transfer request
By providing this value, the TransferManager will not try to
call HeadObject or use the use OS to determine the size of the
transfer.
N( R& ( R R( ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyt provide_transfer_size� s N( R R t __doc__R R R R R R( R R) ( ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR { s R c B s
e Z d Z d d � Z d � Z e d � � Z e d � � Z e d � � Z e d � � Z
d � Z e d � Z
d � Z d
e d � Z d � Z d
� Z d � Z d d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z RS( s* A helper class for managing TransferFuturec C s� | | _ d | _ d | _ d | _ t � | _ g | _ g | _ t j
� | _ t j � | _
t j � | _ t j � | _ t j � | _ d S( Ns not-started( R t _statusR t _resultt
_exceptiont sett _associated_futurest _failure_cleanupst _done_callbackst threadingt Eventt _done_eventt Lockt _lockt _associated_futures_lockt _done_callbacks_lockt _failure_cleanups_lock( R R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR � s c C s d | j j | j f S( Ns %s(transfer_id=%s)( t __class__R R ( R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyt __repr__� s c C s | j S( N( R- ( R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR# � s c C s$ | j � t j | j � SWd QXd S( s� The list of futures associated to the inprogress TransferFuture
Once the transfer finishes this list becomes empty as the transfer
is considered done and there should be no running futures left.
N( R7 t copyR/ ( R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyt associated_futures� s
c C s | j S( s; The list of callbacks to call when the TransferFuture fails( R0 ( R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyt failure_cleanups� s c C s | j S( s� The status of the TransferFuture
The currently supported states are:
* not-started - Has yet to start. If in this state, a transfer
can be canceled immediately and nothing will happen.
* queued - SubmissionTask is about to submit tasks
* running - Is inprogress. In-progress as of now means that
the SubmissionTask that runs the transfer is being executed. So
there is no guarantee any transfer requests had been made to
S3 if this state is reached.
* cancelled - Was cancelled
* failed - An exception other than CancelledError was thrown
* success - No exceptions were thrown and is done.
( R+ ( R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyt status� s c C s/ | j � d | _ | | _ d | _ Wd QXd S( s� Set a result for the TransferFuture
Implies that the TransferFuture succeeded. This will always set a
result because it is invoked on the final task where there is only
ever one final task and it is ran at the very end of a transfer
process. So if a result is being set for this final task, the transfer
succeeded even if something came a long and canceled the transfer
on the final task.
t successN( R6 R R- R, R+ ( R R
( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyt
set_result� s
c C s< | j �- | j � s | r2 | | _ d | _ n Wd QXd S( s� Set an exception for the TransferFuture
Implies the TransferFuture failed.
:param exception: The exception that cause the transfer to fail.
:param override: If True, override any existing state.
t failedN( R6 R R- R+ ( R R# R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR! � s
c C s, | j j t � | j r% | j � n | j S( s� Waits until TransferFuture is done and returns the result
If the TransferFuture succeeded, it will return the result. If the
TransferFuture failed, it will raise the exception associated to the
failure.
( R4 t waitR R- R, ( R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR
� s t c C s | j �p | j � su t } t j d | | � | | � | _ | j d k rV t } n d | _ | ru | j � qu n Wd QXd S( s� Cancels the TransferFuture
:param msg: The message to attach to the cancellation
:param exc_type: The type of exception to set for the cancellation
s %s cancel(%s) calleds not-startedt cancelledN( R6 R t Falset loggert debugR- R+ R" t
announce_done( R t msgt exc_typet should_announce_done( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR s
c C s | j d � d S( s+ Sets the TransferFutrue's status to runningt queuedN( t _transition_to_non_done_state( R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyt set_status_to_queued s c C s | j d � d S( s+ Sets the TransferFuture's status to runningt runningN( RN ( R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyt set_status_to_running! s c C sE | j �6 | j � r2 t d | j | f � � n | | _ Wd QXd S( Ns= Unable to transition from done state %s to non-done state %s.( R6 R t RuntimeErrorR? R+ ( R t
desired_state( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyRN % s
c C s\ t j d | | | j f � | j | d | �} | j | � | j t | j | � � | S( s� Submits a task to a provided executor
:type executor: s3transfer.futures.BoundedExecutor
:param executor: The executor to submit the callable to
:type task: s3transfer.tasks.Task
:param task: The task to submit to the executor
:type tag: s3transfer.futures.TaskTag
:param tag: A tag to associate to the submitted task
:rtype: concurrent.futures.Future
:returns: A future representing the submitted task
s; Submitting task %s to executor %s for transfer request: %s.t tag( RG RH R t submitt add_associated_futuret add_done_callbackR t remove_associated_future( R t executort taskRT t future( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyRU - s
c C s
| j d k S( s� Determines if a TransferFuture has completed
:returns: False if status is equal to 'failed', 'cancelled', or
'success'. True, otherwise
RB RE R@ ( s faileds cancelleds success( R? ( R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR H s c C s$ | j � | j j | � Wd QXd S( s6 Adds a future to be associated with the TransferFutureN( R7 R/ t add( R R[ ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyRV P s
c C s$ | j � | j j | � Wd QXd S( s4 Removes a future's association to the TransferFutureN( R7 R/ t remove( R R[ ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyRX U s
c O s0 | j �! | j j t | | | � � Wd QXd S( s7 Add a done callback to be invoked when transfer is doneN( R8 R1 t appendR ( R t functiont argst kwargs( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyRW Z s
c O s0 | j �! | j j t | | | � � Wd QXd S( s$ Adds a callback to call upon failureN( R9 R0 R^ R ( R R_ R` Ra ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyt add_failure_cleanupa s
c C s7 | j d k r | j � n | j j � | j � d S( sM Announce that future is done running and run associated callbacks
This will run any failure cleanups if the transfer failed if not
they have not been run, allows the result() to be unblocked, and will
run any done callbacks associated to the TransferFuture if they have
not already been ran.
R@ N( R? t _run_failure_cleanupsR4 R. t _run_done_callbacks( R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyRI g s
c C s- | j � | j | j � g | _ Wd QXd S( N( R8 t _run_callbacksR1 ( R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyRd t s
c C s- | j � | j | j � g | _ Wd QXd S( N( R9 Re R> R0 ( R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyRc | s
c C s" x | D] } | j | � q Wd S( N( t
_run_callback( R t callbackst callback( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyRe � s
c C s: y | � Wn( t k
r5 t j d | d t �n Xd S( Ns Exception raised in %s.t exc_info( t ExceptionRG RH R" ( R Rh ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyRf � s
N( R R R* R R R; R R# R= R> R? RA RF R! R
R R RO RQ RN RU R RV RX RW Rb RI Rd Rc Re Rf ( ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR � s2
t BoundedExecutorc B s; e Z e j Z d d d � Z d e d � Z e d � Z RS( c C sR | | _ | d k r! | j } n | d | j � | _ t | � | _ | | _ d S( sD An executor implentation that has a maximum queued up tasks
The executor will block if the number of tasks that have been
submitted and is currently working on is past its maximum.
:params max_size: The maximum number of inflight futures. An inflight
future means that the task is either queued up or is currently
being executed. A size of None or 0 means that the executor will
have no bound in terms of the number of inflight futures.
:params max_num_threads: The maximum number of threads the executor
uses.
:type tag_semaphores: dict
:params tag_semaphores: A dictionary where the key is the name of the
tag and the value is the semaphore to use when limiting the
number of tasks the executor is processing at a time.
:type executor_cls: BaseExecutor
:param underlying_executor_cls: The executor class that
get bounded by this executor. If None is provided, the
concurrent.futures.ThreadPoolExecutor class is used.
t max_workersN( t _max_num_threadsR t EXECUTOR_CLSt _executorR t
_semaphoret _tag_semaphores( R t max_sizet max_num_threadst tag_semaphorest executor_cls( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR � s c C su | j } | r | j | } n | j | j | � } t | j | j | � } t | j j | � � } | j | � | S( s1 Submit a task to complete
:type task: s3transfer.tasks.Task
:param task: The task to run __call__ on
:type tag: s3transfer.futures.TaskTag
:param tag: An optional tag to associate to the task. This
is used to override which semaphore to use.
:type block: boolean
:param block: True if to wait till it is possible to submit a task.
False, if not to wait and raise an error if not able to submit
a task.
:returns: The future assocaited to the submitted task
(
Rp Rq t acquireR R t releaset ExecutorFutureRo RU RW ( R RZ RT t blockt semaphoret
acquire_tokent release_callbackR[ ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyRU � s
c C s | j j | � d S( N( Ro t shutdown( R RC ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR} � s N(
R R R t ThreadPoolExecutorRn R R R" RU R} ( ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyRk � s
%Rx c B s, e Z d � Z d � Z d � Z d � Z RS( c C s
| | _ d S( s� A future returned from the executor
Currently, it is just a wrapper around a concurrent.futures.Future.
However, this can eventually grow to implement the needed functionality
of concurrent.futures.Future if we move off of the library and not
affect the rest of the codebase.
:type future: concurrent.futures.Future
:param future: The underlying future
N( t _future( R R[ ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR � s c C s
| j j � S( N( R R
( R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR
� s c s# � f d � } | j j | � d S( s Adds a callback to be completed once future is done
:parm fn: A callable that takes no arguments. Note that is different
than concurrent.futures.Future.add_done_callback that requires
a single argument for the future.
c s � � S( N( ( t future_passed_to_callback( t fn( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyt
done_callback� s N( R RW ( R R� R� ( ( R� s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyRW � s
c C s
| j j � S( N( R R ( R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR � s ( R R R R
RW R ( ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyRx � s
t BaseExecutorc B s/ e Z d Z d d � Z d � Z e d � Z RS( sA Base Executor class implementation needed to work with s3transferc C s d S( N( ( R Rl ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR s c O s t d � � d S( Ns submit()( R
( R R� R` Ra ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyRU s c C s t d � � d S( Ns
shutdown()( R
( R RC ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR} s N( R R R* R R RU R" R} ( ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR� s t NonThreadedExecutorc B s# e Z d Z d � Z e d � Z RS( s@ A drop-in replacement non-threaded version of ThreadPoolExecutorc O s} t � } y | | | � } | j | � WnM t k
rx t j � d \ } } t j d | | | � | j | | � n X| S( Ni s0 Setting exception for %s to %s with traceback %s( t NonThreadedExecutorFutureRA Rj t sysRi RG RH t set_exception_info( R R� R` Ra R[ R
R t tb( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyRU s
c C s d S( N( ( R RC ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR} s ( R R R* RU R" R} ( ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR� s R� c B sY e Z d Z d � Z d � Z d � Z d d � Z d � Z d � Z d � Z
d � Z RS(
s� The Future returned from NonThreadedExecutor
Note that this future is **not** thread-safe as it is being used
from the context of a non-threaded environment.
c C s1 d | _ d | _ d | _ t | _ g | _ d S( N( R R, R- t
_tracebackRF t _doneR1 ( R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR & s
c C s | | _ | j � d S( N( R, t _set_done( R R
( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyRA - s c C s | | _ | | _ | j � d S( N( R- R� R� ( R R# t traceback( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR� 1 s c C s5 | j r. t j t | j � | j | j � n | j S( N( R- R t reraiset typeR� R, ( R t timeout( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR
6 s c C s7 t | _ x | j D] } | j | � q Wg | _ d S( N( R" R� R1 t _invoke_done_callback( R R� ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR� < s c C s
| | � S( N( ( R R� ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR� B s c C s | j S( N( R� ( R ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR E s c C s- | j r | j | � n | j j | � d S( N( R� R� R1 R^ ( R R� ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyRW H s N( R R R* R RA R� R R
R� R� R RW ( ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyR� s t TaskTagt namet in_memory_uploadt in_memory_download(" t
concurrentR t collectionsR R< t loggingR� R2 t s3transfer.compatR R t s3transfer.exceptionsR R t s3transfer.utilsR R t getLoggerR RG t objectR R R R R Rk Rx R� R� R� R� t IN_MEMORY_UPLOAD_TAGt IN_MEMORY_DOWNLOAD_TAG( ( ( s6 /usr/lib/python2.7/site-packages/s3transfer/futures.pyt <module>
s0 1&�L# /