U 9%e @sUddlZddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z m Z mZmZddlmZddlmZmZddlZddlZdddd d d d d dddg ZdZeedfeejdfddd Ze edddd ZdJddZGdddZddZee ee ejfddd Zdddd Z dKdd Z!d!d"Z"Gd#d d ej#j$Z%d$dZ&ej'de&ed%d&ee(egee e ffe)e(d&d'dZ*dLd)dZ+d*d+Z,d(a-ej.e(d,d-dZ/Gd.d/d/Z0Gd0d1d1Z1Gd2d3d3ej#j$Z2Gd4d5d5Z3d6Z4Gd7dde5Z6eegefee6gdffdd8d9Z7eje e)efd:d;d<Z8ee8d=d>d?iZ9e e)eejgeffe:d@<GdAdBdBe;ZZ?GdEdFdFej#j=j>Z@d(e&ed%fegee e ffe)e(dGdHdIZAdS)MN) defaultdict) AnyCallableContextManager DefaultDictDictIterableListOptionalTuple) ReferenceType)LoggingTensorMode capture_logs checkpointcheckpoint_sequentialCheckpointErrorCheckpointFunctioncheck_backward_validitydetach_variableget_device_statesset_device_statesnoop_context_fnset_checkpoint_early_stopDefaultDeviceTypedefault.)inputsreturncCsft|trRg}|D]6}t|tjs.||q|}|j|_||qt|Stdt|j dS)Nz@Only tuple of tensors is supported. Got Unsupported input type: ) isinstancetupletorchTensorappenddetach requires_grad RuntimeErrortype__name__)routinpxr*U/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/torch/utils/checkpoint.pyr(s    cCs tdd|DstddS)Ncss |]}t|tjr|jVqdSN)rrr r#.0r(r*r*r+ <s z*check_backward_validity..zBNone of the inputs have requires_grad=True. Gradients will be None)anywarningswarn)rr*r*r+r;scudacCstt|}|Sr,)getattrrdevice device_moduler*r*r+_get_device_moduleBs r8c@s:eZdZdZdZed edddZeedddZd S) ra@ A class that manages the default device type for checkpointing. If no non-CPU tensors are present, the default device type will be used. The default value is 'cuda'. The device type is used in the checkpointing process when determining which device states to save and restore for recomputation. r3r6cCs |t_dS)z Set the default device type for checkpointing. Args: device (str): The device type to be set as default. Default is 'cuda'. Nr_default_device_typer9r*r*r+set_device_typeQsz!DefaultDeviceType.set_device_typercCstjS)z Get the current default device type for checkpointing. Returns: str: The current default device type. r:r*r*r*r+get_device_type[sz!DefaultDeviceType.get_device_typeN)r3) r& __module__ __qualname____doc__r; staticmethodstrr<r>r*r*r*r+rGs  cGsTtdd|D}t|dkr(tdt|dkrhs z%_infer_device_type..aTensor arguments, excluding CPU tensors, are detected on at least two types of devices. Device state will only be saved for devices of a single device type, and the remaining devices will be ignored. Consequently, if any checkpointed functions involve randomness, this may result in incorrect gradients. (Note that if CUDA devices are among the devices detected, it will be prioritized; otherwise, the first device encountered will be selected.)rr3)listlenr1r2rr>)argsZ device_typesr*r*r+_infer_device_typefs  rMr=c GsXtdd|D}g}tt|}|D](}||||W5QRXq&||fS)NcSs,h|]$}t|tjr|jjdks|qSrD)rrr r6r%Z get_devicerFr*r*r+rHs z$get_device_states..)rJr8rMr6r! get_rng_state)rLZfwd_device_idsfwd_device_statesr7Z device_idr*r*r+rs  c CsDtt|}t||D](\}}||||W5QRXqdSr,)r8rMzipr6 set_rng_state)devicesZstatesr7r6stater*r*r+rs  cCsp|dkr"tttd}n.t|rLt|}||td}nd}tttd}||fS)Nr3)enableddtypeZ cache_enabled) ris_autocast_enabledZget_autocast_gpu_dtypeZis_autocast_cache_enabled_supports_autocastr8get_autocast_dtypeZis_autocast_cpu_enabledZget_autocast_cpu_dtype)r6device_autocast_kwargsr7cpu_autocast_kwargsr*r*r+_get_autocast_kwargss"r[cCs$t|}|dkp"t|do"t|dS)Nr3rVrX)r8hasattrr5r*r*r+rWsrWc@s$eZdZeddZeddZdS)rc Gst|||_||_t||_t|j\|_|_|rpt |_ d|_ t |j}t |ddrpd|_ t|\|_|_g|_g|_g}t|D]B\}}t|r|||j||jdq|j|q|j|t||}W5QRX|S)NF _initializedT)r run_functionpreserve_rng_staterMr6r[rYrZrrN fwd_cpu_statehad_device_in_fwdr8r4r fwd_devicesrOrtensor_indices enumerate is_tensorr!save_for_backwardno_grad) ctxr^r_rLr7Z tensor_inputsirGoutputsr*r*r+forwards6          zCheckpointFunction.forwardc Gstjstdt|j}|j}|j}t|j }t |D]\}}||||<q:g}|j rf|j rf|j }tjj||j |j d|j rt|j|j rt|j |jtt|} t|j r|jjf|jnt} t>| .tjjjf|j|j| } W5QRXW5QRXW5QRXW5QRXt | tj!r<| f} g} g} t"t#| D]>}t$| |rP| |j%rP| &| || &||qPt#| dkrtdtj'| | tdd| D}d|S)NzCheckpointing is not compatible with .grad() or when an `inputs` parameter is passed to .backward(). Please use .backward() and do not pass its `inputs` argument.rRrTZ device_typerzInone of output has requires_grad=True, this checkpoint() is not necessarycss$|]}t|tjr|jndVqdSr,)rrr Zgradr-r*r*r+r/!sz.CheckpointFunction.backward..)NN)(rautogradZ_is_checkpoint_validr$rJrrc saved_tensorsr8r6rdr_rarbrandomfork_rngrQr`rrOrrrWampautocastrY contextlib nullcontext enable_gradrErZr^rr rangerKrer#r!backward)rhrLrrctensorsr7riidx rng_devicesZdetached_inputsdevice_autocast_ctxrjZoutputs_with_gradZargs_with_gradZgradsr*r*r+rwsd       2zCheckpointFunction.backwardN)r&r?r@rBrkrwr*r*r*r+rs (cCsttfSr,)rsrtr*r*r*r+r)sF) use_reentrant context_fndeterminism_checkdebugc Os|dkrtdd}|dd}|rF|rFtdddd|D|rt|tk sZ|d k rbtd tj||f|St|||||f||}t ||||} z t |Wnt k r| YSXdS) ahCheckpoint a model or part of the model Activation checkpointing is a technique that trades compute for memory. Instead of keeping tensors needed for backward alive until they are used in gradient computation during backward, forward computation in checkpointed regions omits saving tensors for backward and recomputes them during the backward pass. Activation checkpointing can be applied to any part of a model. There are currently two checkpointing implementations available, determined by the :attr:`use_reentrant` parameter. It is recommended that you use ``use_reentrant=False``. Please refer the note below for a discussion of their differences. .. warning:: If the :attr:`function` invocation during the backward pass differs from the forward pass, e.g., due to a global variable, the checkpointed checkpointed version may not be equivalent, potentially causing an error being raised or leading to silently incorrect gradients. .. warning:: If you are using the ``use_reentrant=True`` variant (this is currently the default), please refer to the note below for important considerations and potential limitations. .. note:: The reentrant variant of checkpoint (``use_reentrant=True``) and the non-reentrant variant of checkpoint (``use_reentrant=False``) differ in the following ways: * Non-reentrant checkpoint stops recomputation as soon as all needed intermediate activations have been recomputed. This feature is enabled by default, but can be disabled with :func:`set_checkpoint_early_stop`. Reentrant checkpoint always recomputes :attr:`function` in its entirety during the backward pass. * The reentrant variant does not record the autograd graph during the forward pass, as it runs with the forward pass under :func:`torch.no_grad`. The non-reentrant version does record the autograd graph, allowing one to perform backward on the graph within checkpointed regions. * The reentrant checkpoint only supports the :func:`torch.autograd.backward` API for the backward pass without its `inputs` argument, while the non-reentrant version supports all ways of performing the backward pass. * At least one input and output must have ``requires_grad=True`` for the reentrant variant. If this condition is unmet, the checkpointed part of the model will not have gradients. The non-reentrant version does not have this requirement. * The reentrant version does not consider tensors in nested structures (e.g., custom objects, lists, dicts, etc) as participating in autograd, while the non-reentrant version does. * The reentrant checkpoint does not support checkpointed regions with detached tensors from the computational graph, whereas the non-reentrant version does. For the reentrant variant, if the checkpointed segment contains tensors detached using ``detach()`` or with :func:`torch.no_grad`, the backward pass will raise an error. This is because ``checkpoint`` makes all the outputs require gradients and this causes issues when a tensor is defined to have no gradient in the model. To avoid this, detach the tensors outside of the ``checkpoint`` function. Args: function: describes what to run in the forward pass of the model or part of the model. It should also know how to handle the inputs passed as the tuple. For example, in LSTM, if user passes ``(activation, hidden)``, :attr:`function` should correctly use the first input as ``activation`` and the second input as ``hidden`` preserve_rng_state(bool, optional): Omit stashing and restoring the RNG state during each checkpoint. Default: ``True`` use_reentrant(bool, optional): Use checkpointing implementation that requires re-entrant autograd. If ``use_reentrant=False`` is specified, ``checkpoint`` will use an implementation that does not require re-entrant autograd. This allows ``checkpoint`` to support additional functionality, such as working as expected with ``torch.autograd.grad`` and support for keyword arguments input into the checkpointed function. Note that future versions of PyTorch will default to ``use_reentrant=False``. Default: ``True`` context_fn(Callable, optional): A callable returning a tuple of two context managers. The function and its recomputation will be run under the first and second context managers respectively. This argument is only supported if ``use_reentrant=False``. determinism_check(str, optional): A string specifying the determinism check to perform. By default it is set to ``"default"`` which compares the shapes, dtypes, and devices of the recomputed tensors against those the saved tensors. To turn off this check, specify ``"none"``. Currently these are the only two supported values. Please open an issue if you would like to see more determinism checks. This argument is only supported if ``use_reentrant=False``, if ``use_reentrant=True``, the determinism check is always disabled. debug(bool, optional): If ``True``, error messages will also include a trace of the operators ran during the original forward computation as well as the recomputation. This argument is only supported if ``use_reentrant=False``. args: tuple containing inputs to the :attr:`function` Returns: Output of running :attr:`function` on :attr:`*args` Naatorch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.Tr_Unexpected keyword arguments: ,css|] }|VqdSr,r*rFr*r*r+r/szcheckpoint..FzKPassing `context_fn` or `debug` is only supported when use_reentrant=False.) r1r2pop ValueErrorjoinrrapply'_checkpoint_without_reentrant_generatornext StopIteration) functionr|r}r~rrLkwargspreservegenretr*r*r+r6sBv   Tc Ks|dd}|r,tdddd|Ddd}t|tjjrNt|}t ||}d }t d ||d |D](} | |d }t || |||||d }qr||d t |d ||S) a%A helper function for checkpointing sequential models. Sequential models execute a list of modules/functions in order (sequentially). Therefore, we can divide such a model in various segments and checkpoint each segment. All segments except the last will not store the intermediate activations. The inputs of each checkpointed segment will be saved for re-running the segment in the backward pass. .. warning:: If you are using the ``use_reentrant=True` variant (this is the default), please see :func:`~torch.utils.checkpoint.checkpoint` for the important considerations and limitations of this variant. It is recommended that you use ``use_reentrant=False``. .. warning: Since PyTorch 1.4, it allows only one Tensor as the input and intermediate outputs, just like :class:`torch.nn.Sequential`. Args: functions: A :class:`torch.nn.Sequential` or the list of modules or functions (comprising the model) to run sequentially. segments: Number of chunks to create in the model input: A Tensor that is input to :attr:`functions` preserve_rng_state(bool, optional): Omit stashing and restoring the RNG state during each checkpoint. Default: ``True`` use_reentrant(bool, optional): Use checkpointing implementation that requires re-entrant autograd. If ``use_reentrant=False`` is specified, ``checkpoint`` will use an implementation that does not require re-entrant autograd. This allows ``checkpoint`` to support additional functionality, such as working as expected with ``torch.autograd.grad`` and support for keyword arguments input into the checkpointed function. Default: ``True`` Returns: Output of running :attr:`functions` sequentially on :attr:`*inputs` Example: >>> # xdoctest: +SKIP("stub") >>> model = nn.Sequential(...) >>> input_var = checkpoint_sequential(model, chunks, input_var) r_Trrcss|] }|VqdSr,r*rFr*r*r+r/sz(checkpoint_sequential..csfdd}|S)Ncs$tdD]}||}q|S)NrI)rv)inputjend functionsstartr*r+rksz.run_function..forwardr*)rrrrkr*rr+r^sz+checkpoint_sequential..run_functionrrI)r|r_) rrrrrnnZ SequentialrJchildrenrKrvr) rsegmentsrr|rrr^Z segment_sizerrr*r*r+rs&-     cCs|s tddS)NzqSomething went unexpectedly wrong in activation checkpoint. Please report this bug by filing an issue to PyTorch.AssertionError)Zcondr*r*r+_internal_assertsr)enableccszt}|adVW5|aXdS)a-Context manager that sets whether checkpoint should stop recomputation early. By default, non-reentrant checkpoint stops recomputation as soon as it has computed all needed Tensors. This context manager can be used to disable that feature if it is problematic for your specific application. This context manager only needs to be active when forward is run. It does not need to be active during backward. Example:: >>> # xdoctest: +SKIP(failing) >>> message = "saved tensors default hooks are disabled" >>> with set_checkpoint_early_stop(False): ... # Any checkpoint under this context manager will respect this ... # context manager, even if its backward is performed outside. ... out = checkpoint(fn, inputs) ... >>> out.backward() N)_enable_checkpoint_early_stop)rprevr*r*r+rs  c@s eZdZdS)_HandleNr&r?r@r*r*r*r+rsrc@seZdZddZdS)_HoldercCs t|_dSr,)dicthandlesselfr*r*r+__init__sz_Holder.__init__N)r&r?r@rr*r*r*r+rsrc@sDeZdZeddZeeeedfeddddZedd ZdS) _NoopSaveInputscGs tdS)Nr)rempty)rLr*r*r+rksz_NoopSaveInputs.forward.N)rhroutputrcs^tddt|D\}ddtDdd|Dfdd}||_|j|dS)NcSs$g|]\}}t|tjr||fqSr*rrr r.rior*r*r+ s z1_NoopSaveInputs.setup_context..cSsi|]\}}||qSr*r*)r.abr*r*r+ sz1_NoopSaveInputs.setup_context..cSs g|]}t|tjrdn|qSr,r)r.rr*r*r+rscs&fddtD}|ddS)Ncs(g|] \}}|kr |n|qSr*r*r) idx2saved_idxrnrcr*r+rszC_NoopSaveInputs.setup_context..get_args..rI)rd)rnrrLrrc)rnr+get_argssz/_NoopSaveInputs.setup_context..get_args)rPrdrrf)rhrrrxrr*rr+ setup_contexts z_NoopSaveInputs.setup_contextcGs tddS)Nz(Did not expect to backward on this graphr)rhZ grad_outputsr*r*r+rwsz_NoopSaveInputs.backward) r&r?r@rBrkrr rrwr*r*r*r+rs  rc@seZdZddZddZdS)_CheckpointFramecCsZ||_d|_g|_ttj|_tt|_tt |_ ||_ ||_ ||_ g|_d|_d|_dS)NF) recompute_fn input_saver weak_holdersrweakrefWeakKeyDictionary recomputedintrecomp_counterbool is_recomputed early_stop metadata_fnunpack_error_cb x_metadatasforward_completedignore_saved_mismatch)rrrrrr*r*r+rs  z_CheckpointFrame.__init__c Cs0|jr dSt|j|j|ks>tdt|jd|j|g}t|jD]\}}|}|dkrdqLt||jkt|j|dk t|j||j|k|j |}|j||j|}|| |krL| ||| |fqLt|dkr,d}|D]&\}}} |d|d|d| d7}qtd |dS) Nztorch.utils.checkpoint: A different number of tensors was saved during the original forward and recomputation. Number of tensors saved during forward: z/ Number of tensors saved during recomputation: rztensor at position z: saved metadata: z recomputed metadata:  zztorch.utils.checkpoint: Recomputed values for the following tensors have different metadata than during the forward pass. ) rrKrrrrdrrrrrr!) rgidZnb_meta_differentryZ weak_holderholderZx_metaZ recomputed_xZmismatched_tensorsZrecomputed_metar*r*r+check_recomputed_tensors_matchs6  z/_CheckpointFrame.check_recomputed_tensors_matchN)r&r?r@rrr*r*r*r+rsra An error happened while unpacking tensors; dumping logs of latest computation because you passed `debug=True` to `torch.utils.checkpoint.checkpoint()`. Scroll all the way down for guidance on how to navigate these logs. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+ | 1. Stack traces of the operators that ran in the original forward | +------------------------------------------------------------------------------+ {forward_traces} +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+ | 2. Stack traces of the operators that ran during recomputation | +------------------------------------------------------------------------------+ {recompute_traces} +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+ | 3. Log of operators in the original forward and recomputation | +------------------------------------------------------------------------------+ (Scroll up to correlate stack traces with each operation listed below. This helps identify their source in the code.) IMPORTANT: Differences in "detach" calls between the original forward and the recomputation are expected. They are introduced by the checkpointing mechanism and can be ignored. Operations executed during the original forward: {forward_ops} Operations executed during recomputation: {recompute_ops} +------------------------------------------------------------------------------+ ERROR: Detected non-determinism while running activation checkpointing You are seeing this error because you passed `debug=True` to checkpoint and tensors to be saved during the original forward and differ between those saved during recomputation. This can happen if different operators were ran in the original forward and in the recomputation. To identify where the mismatch may be coming from, you can do the following: 1) Compare the operators ran during original forward and recomputation to see where they differ. These operators are printed above in the order they were executed. 2) Review the stack trace for each operator to locate its invocation source. Each operator's stack trace is printed in their execution order. Note that the logs can be quite long. Here's how they are structured: (Tip: you can Ctrl-f for these headers) 1. Stack traces of the operators that ran in the original forward 2. Stack traces of the operators that ran during recomputation 3. Log of operators in the original forward and recomputation 4. Error message <--- You are here -------------------------------------------------------------------------------- c@s eZdZdS)rNrr*r*r*r+rscs`tdkotdkGfddd}||tdfdd }fdd }||fS) Nx86_64Linuxcs eZdZddZfddZdS)z._get_debug_context_and_cb..CaptureLogscSsd|_d|_dSr,)logstbsrr*r*r+rsz7_get_debug_context_and_cb..CaptureLogs.__init__cstjfdd}|S)Nc 3sDt4tdddd}|\__|VW5QRXW5QRXdS)NT)Z python_tbZ script_tbcpp_tb)r rrr)Z logs_and_tb)rrr*r+ logging_modes  zX_get_debug_context_and_cb..CaptureLogs.get_context_manager..logging_mode)rscontextmanager)rrrrr+get_context_managerszB_get_debug_context_and_cb..CaptureLogs.get_context_managerN)r&r?r@rrr*rr*r+ CaptureLogssr)ecs\dd}jdk stjdk s$tttj|d|ddjdjd|dS)Nc Ssd}t|j}tt|j|jD]\}\}}||d|dd|d|d7}d}|D]L}|dd k} |st| stqXn | r~d }qX||d d |d d |dd7}qX|d7}q |S)Nrz (rIz of z in z) FnameZ__torch_dispatch__Tfilename:linerz )rKrrdrPr) labelrr'Z total_lenrilogtbZfound_torch_dispatchrZis_torch_dispatchr*r*r+ get_str_tbs $ ( zF_get_debug_context_and_cb..unpack_error_cb..get_str_tboriginalZ recomputer)Zforward_tracesZrecompute_tracesZ forward_opsZ recompute_ops)rrr_checkpoint_error_templateformatr)rrcapture_logs_fwdcapture_logs_recomputer*r+rs  z2_get_debug_context_and_cb..unpack_error_cbcsfSr,)rr*rr*r+r}sz-_get_debug_context_and_cb..context_fn)platformmachinesystemr)rrr}r*)rrrr+_get_debug_context_and_cbsr)r)rcCs|j|j|jdS)NshaperUr6rr)r*r*r+_default_meta_extractorsrnonecCsdSr,r*)_r*r*r+r"_allowed_determinism_checks_to_fnsc@s eZdZdS)_StopRecomputationErrorNrr*r*r*r+rsrcs$eZdZeedfdd ZZS)_recomputation_hook)target_frame_refrcs(fdd}dd}t||dS)Ncs}|dk st|j}|jd7<|t|jkrb|jrFt|jsZd|_|Std|j|}|dk rt |j ddkt |j <||j |j <|jr|jt|jkrt|S)NrITzotorch.utils.checkpoint: trying to save more tensors during recomputation than during the original forward pass.)rrrKrrrrr"rrrgetrrr)r)Z target_frameZ recomp_idxrrrr*r+ pack_hooks,     z/_recomputation_hook.__init__..pack_hookcSs|Sr,r*rr*r*r+ unpack_hook sz1_recomputation_hook.__init__..unpack_hook)superr)rrrrr __class__rr+rs%z_recomputation_hook.__init__)r&r?r@r rr __classcell__r*r*rr+rsrcseZdZfddZZS)_checkpoint_hookcsRfdd}fddjdk r@fdd}t||nt|dS)Nc sLt}jt|jdk rHtj|W5QRX|Sr,) rrr!rrefrrrgr)r)rframer*r+rs   z,_checkpoint_hook.__init__..pack_hookc stj}|dkrtt}j|sjj}| |j }z@t t |&tjj|W5QRXW5QRXWntk rYnXdj|<|t||jk|j|dkrtdt|j|j|kj||j|}d|j|<|S)NrTztorch.utils.checkpoint: Unpack is being triggered for a tensor that was already unpacked once. If you are calling ctx.saved_tensors in backward, make sure to do so only once. Otherwise please open an issue with details on your use case.)rZ_CZ_current_graph_task_idruuiduuid4rrgrad_fnrrnrrrrmrurrrrrrr)rrrhrLrrr*r+r!s6    "   z._checkpoint_hook.__init__..unpack_hookc s<z |WStk r6}z|W5d}~XYnXdSr,)rr)rrrrr*r+unpack_hook_with_error_cbCs z<_checkpoint_hook.__init__..unpack_hook_with_error_cb)rrr)rrrrrrr+rs  ! z_checkpoint_hook.__init__)r&r?r@rrr*r*rr+rsr)r}r~rc /sbd}|r"|tkrtdt\}}|tkr4t|}ntdttd|t|t|\} td\ rt dt ddrdt |\ f d d } t | t||} t jd dd } tj| |f|| _| jjdkr dVdSt| | dVW5QRXW5QRXd| _t ddr^ r^s^td dS)aXCheckpointing without reentrant autograd Args: function: describes what to run in the forward pass of the model or part of the model. It should also know how to handle the inputs passed as the tuple. For example, in LSTM, if user passes ``(activation, hidden)``, :attr:`function` should correctly use the first input as ``activation`` and the second input as ``hidden`` preserve_rng_state(bool, optional): Omit stashing and restoring the RNG state during each checkpoint. Default: ``True`` context_fn(Callable, optional): A callable returning a tuple of two context managers. The function and its recomputation will be run under the first and second context managers respectively. determinism_check(str, optional): A string specifying the determinism check to perform. By default it is set to ``"default"`` which compares the shapes, dtypes, and devices of the recomputed tensors against those the saved tensors. To turn off this check, specify ``"none"``. Currently these are the only two supported values. Please open an issue if you would like to see more determinism checks. debug(bool, optional): If ``True``, error messages will also include a trace of the operators ran during the original forward computation as well as the recomputation. *args: Arguments to pass in to the given ``function``. **kwargs: Keyword arguments to pass into the given ``function``. Nz6debug=True is incompatible with non-default context_fnz#determinism_check should be one of z , but got r9Fr]Tc s|^}}g} rr}tjj| d rHtrHttr^jjfnt }|<tj jjf  ||W5QRXW5QRXW5QRXW5QRXdS)Nrl) rrorprQrrWrqrrrsrtrE)rrrLrzr{ rZr6rYr7fnr`rOrbrar_Zrecompute_contextr*r+rs.   z=_checkpoint_without_reentrant_generator..recompute_fnr)r#zPyTorch's device state was initialized in the forward pass of a Checkpoint, which is not allowed. Please open an issue if you need this feature.)rrrrrJkeysrMr8r[rrNr4rrrrrrrrrrr$) rr_r}r~rrLrrrZforward_contextrZ new_framedummyr*rr+rPs\#      r)r3)r3)T)Brsrr1r collectionsrtypingrrrrrrr r r r Z&torch.testing._internal.logging_tensorr rrr__all__Z_DEFAULT_DETERMINISM_MODEr rrr8rrMrrrr[rWrmFunctionrrZ_disable_dynamorrCrrrrrrrrrrrr$rrrr__annotations__ ExceptionrgraphZsaved_tensors_hooksrrrr*r*r*r+s , "   j  L %_<(;  /=