U 0-e@sHddlmZddlZddlZddlZddlZddlmZddlm Z ddl m Z m Z m Z ddlZddlmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZddlm Z ed d rddl!m"m#Z$ed d rddl%Z%d d d dZ&ddZ'Gdddej(Z)ed d se*ne)Z+GdddZ,GdddZ-GdddZ.dS)) annotationsN)contextmanager)partial)AnyCallableOptional)DistributedType DynamoBackendGradientAccumulationPluginget_ccl_versionget_int_from_envis_ccl_availableis_deepspeed_availableis_fp8_availableis_ipex_availableis_mps_availableis_npu_availableis_tpu_availableis_xpu_availableparse_choice_from_envparse_flag_from_env)SageMakerDistributedTypeF)Z check_deviceboolreturncCs tjikS)z Checks if the `AcceleratorState` has been initialized from `Accelerator`. Same as `AcceleratorState.initialized`, but works as a module method. )AcceleratorState _shared_staterrQ/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/accelerate/state.pyis_initialized7sr cOsdSNr)argskwargsrrr do_nothing@sr$c@s2eZdZdZd ddddZd dd Zd d ZdS)ThreadLocalSharedDicta Descriptor that holds a dict shared between instances of a class in the same thread. Note: Descriptors have slightly different semantics than just a dict field on its own. `PartialState(...)._shared_state` and `PartialState._shared_state` (instance vs class) give the same value: the underlying _storage dict. Likewise, `PartialState(...)._shared_state = {...}` overrides the _storage dict inside the descriptor as you would expect. However, `PartialState._shared_state = {}` actually replaces the descriptor object with a dict instead Thus, you should modify the _storage dict in-place (e.g. `_shared_state.clear()`). See Python documentation for an explanation of descriptors: https://docs.python.org/3/howto/descriptor.html This is required for using PyTorch/XLA with PJRT in multithreaded mode (required for TPU v2 and v3). See https://github.com/pytorch/xla/blob/r2.0/docs/pjrt.md#multithreading-on-tpu-v2v3 Fr) thread_localcCs i|_dSr!Z_storage)selfr&rrr__init__UszThreadLocalSharedDict.__init__NcCs|jSr!r')r(objobjtyperrr__get__XszThreadLocalSharedDict.__get__cCs ||_dSr!r')r(r*valuerrr__set__[szThreadLocalSharedDict.__set__)F)N)__name__ __module__ __qualname____doc__r)r,r.rrrrr%Ds r%c@sDeZdZdZeZd9ddddZddd d Zed d Z e ddd dZ e ddZ e ddddZ e ddddZe ddddZddZddddZed:dddddZed d!Zed"d#Zd;d%d&d'd(Zdd%d-d1d2d3Zd4d5Ze d6dd7d8Zd$S)? PartialStateah Singleton class that has information about the current training environment and functions to help with process control. Designed to be used when only process control and device execution states are needed. Does *not* need to be initialized from `Accelerator`. **Available attributes:** - **device** (`torch.device`) -- The device to use. - **distributed_type** ([`~accelerate.state.DistributedType`]) -- The type of distributed environment currently in use. - **local_process_index** (`int`) -- The index of the current process on the current server. - **mixed_precision** (`str`) -- Whether or not the current script will use mixed precision, and if so the type of mixed precision being performed. (Choose from 'no','fp16','bf16 or 'fp8'). - **num_processes** (`int`) -- The number of processes currently launched in parallel. - **process_index** (`int`) -- The index of the current process. - **is_last_process** (`bool`) -- Whether or not the current process is the last one. - **is_main_process** (`bool`) -- Whether or not the current process is the main one. - **is_local_main_process** (`bool`) -- Whether or not the current process is the main one on the local node. - **debug** (`bool`) -- Whether or not the current script is being run in debug mode. Fr)cpucKs|j|_|js||_d|_tjdd}|dk rdd3| }|dkrd}t?|t@Ad4|d5tjs:| dd| |_tjj|jf| | d6|tj|_tj|_|rft d7|_ n0t+rt d|j|_ tj.|j n|jB|_ n>tj|_d|_d|_|_|j dkr|rt d7n|jB|_ t d8d|_CdS)9NZACCELERATE_TORCH_DEVICEZACCELERATE_DEBUG_MODEZ_use_sagemaker_dpZACCELERATE_USE_SAGEMAKERfalsetrueZ%ACCELERATE_SAGEMAKER_DISTRIBUTED_TYPErZsmddp)backendZ LOCAL_RANKcudaACCELERATE_USE_DEEPSPEEDz_DeepSpeed is not available => install it using `pip3 install deepspeed` or build it from source)commr7ZcclZncclF)Z dist_backendZauto_mpi_discoveryxpunoZhcclnpuZPMI_SIZEZOMPI_COMM_WORLD_SIZEZMV2_COMM_WORLD_SIZEZ WORLD_SIZErZCCL_WORKER_COUNTz1.12ZmpiZglooZRANKZPMI_RANKZOMPI_COMM_WORLD_RANKZMV2_COMM_WORLD_RANKZMPI_LOCALRANKIDZOMPI_COMM_WORLD_LOCAL_RANKZMV2_COMM_WORLD_LOCAL_RANKZMPI_LOCALNRANKSZOMPI_COMM_WORLD_LOCAL_SIZEZMV2_COMM_WORLD_LOCAL_SIZEZ MASTER_PORTZ29500Z MASTER_ADDRzwLooks like distributed multinode run but MASTER_ADDR env not set, please try exporting rank 0's hostname as MASTER_ADDRZOMP_NUM_THREADSZMKL_NUM_THREADS)Zlogicalz4OMP_NUM_THREADS/MKL_NUM_THREADS unset, we set it at z to improve oob performance.)rankZ world_sizer4Z FORK_LAUNCHED)Dr__dict__ initializedZ_cpur7osenvirongettorchdevicerdebugpoprNOZ DATA_PARALLELr MULTI_GPUdistributed_typeZ,smdistributed.dataparallel.torch.torch_smddp distributedr Zinit_process_groupZget_world_size num_processesZget_rank process_indexintlocal_process_indexr9Z set_devicerTPUxmZxrt_world_sizeZ get_ordinalZget_local_ordinalZ xla_devicerAssertionError DEEPSPEEDZ deepspeedr;rrZinit_distributedr<_mixed_precision is_availabler MULTI_NPUr>r MULTI_XPU MULTI_CPUr oneccl_bindings_for_pytorch torch_cclZis_mpi_availablestr ValueErrorpsutil cpu_countZset_num_threadswarningswarndefault_deviceZ fork_launched)r(r4r#Z env_deviceZuse_sagemaker_dpZ smdistributeddistrZr[r7r?sizeZ local_rankZ local_sizer^Znum_cpu_threads_per_processrrrr)|s8                   *     &                       zPartialState.__init__r\rc CsBd|j|jrd|jndd|jd|jd|jd|jd S) NzDistributed environment: z Backend: z Num processes: z Process index: z Local process index: z Device:  )rKr7rMrNrPrFr(rrr__repr__+s@zPartialState.__repr__cCstjdSzCResets `_shared_state`, is used internally and should not be calledN)r3rclearrrrr _reset_state4szPartialState._reset_statecCs |jikS)z7Returns whether the `PartialState` has been initialized)rrgrrrrA9szPartialState.initializedcCs|jtjko|jdkS)P Whether the Accelerator is configured for distributed training r)rKr rIrMrgrrruse_distributed>szPartialState.use_distributedcCs|j|jdkS)3Returns whether the current process is the last oner)rNrMrgrrris_last_processEszPartialState.is_last_processcCs|jtjkr|jdkS|jS)7Returns whether the current process is the main processr)rKr MEGATRON_LMrNrorgrrris_main_processJszPartialState.is_main_processcCs|jtjkr|jdkS|jS)IReturns whether the current process is the main process on the local noder)rKr rqrProrgrrris_local_main_processQs z"PartialState.is_local_main_processcCsH|jtjtjtjtjtjtjfkr.tj n|jtj krDt ddS)a Will stop the execution of the current process until every other process has reached that point (so this does nothing when the script is only run in one process). Useful to do before saving a model. Example: ```python >>> # Assuming two GPU processes >>> import time >>> from accelerate.state import PartialState >>> state = PartialState() >>> if state.is_main_process: ... time.sleep(2) >>> else: ... print("I'm waiting for the main process to finish its sleep...") >>> state.wait_for_everyone() >>> # Should print on every process at the same time >>> print("Everyone is here") ``` z"accelerate.utils.wait_for_everyoneN)rKr rJrWrXrYrTFSDPrErLZbarrierrQrRZ rendezvousrgrrrwait_for_everyoneZs  zPartialState.wait_for_everyone)is_mainccs"|s |dV|r|dSr!)rv)r(rwrrr _goes_first|s zPartialState._goes_first"list | tuple | dict | torch.Tensorinputs apply_paddingc#sjdkr|VdSt|t|tr`t|t|dtfdd|Ds`tdt jj }|}t|jdkrj jdkr}fdd|||VdS) a} Splits `input` between `self.num_processes` quickly and can be then used on that process. Useful when doing distributed inference, such as with different prompts. Note that when using a `dict`, all keys need to have the same number of elements. Args: inputs (`list`, `tuple`, `torch.Tensor`, or `dict` of `list`/`tuple`/`torch.Tensor`): The input to split between processes. apply_padding (`bool`, `optional`, defaults to `False`): Whether to apply padding by repeating the last element of the input so that all processes have the same number of elements. Useful when trying to perform actions such as `gather()` on the outputs or passing in less inputs than there are processes. If so, just remember to drop the padded elements afterwards. Example: ```python # Assume there are two processes from accelerate import PartialState state = PartialState() with state.split_between_processes(["A", "B", "C"]) as inputs: print(inputs) # Process 0 ["A", "B"] # Process 1 ["C"] with state.split_between_processes(["A", "B", "C"], apply_padding=True) as inputs: print(inputs) # Process 0 ["A", "B"] # Process 1 ["C", "C"] ``` rNrc3s|]}t|kVqdSr!)len).0v)lengthrr sz7PartialState.split_between_processes..z6All values in the dictionary must have the same lengthcst|tttjfr|t|kr,|dd}n |||}rt|tjrvddlm}m}||j }|||dd}n||dgt|7}|St|t r| D]}||||||<q|S|SdS)Nr8r)pad_across_processessend_to_device)Z pad_index) isinstancelisttuplerEZTensorr}Zaccelerate.utilsrrrFdictkeys)r{ start_index end_indexresultrrZtensorized_resultkey) _split_valuesr|num_samples_per_processr(rrrs       z;PartialState.split_between_processes.._split_values) rMr}rrrrallvaluesr]mathceilrN)r(r{r|rrr)rr|rrr(rsplit_between_processess'   "z$PartialState.split_between_processesccs||jEdHdS)a1 Lets the main process go first inside a with block. The other processes will enter the with block after the main process exits. Example: ```python >>> from accelerate import Accelerator >>> accelerator = Accelerator() >>> with accelerator.main_process_first(): ... # This will be printed first by process 0 then in a seemingly ... # random order by the other processes. ... print(f"This will be printed by process {accelerator.process_index}") ``` N)rxrrrgrrrmain_process_firstszPartialState.main_process_firstccs||jEdHdS)a9 Lets the local main process go inside a with block. The other processes will enter the with block after the main process exits. Example: ```python >>> from accelerate.state import PartialState >>> state = PartialState() >>> with state.local_main_process_first(): ... # This will be printed first by local process 0 then in a seemingly ... # random order by the other processes. ... print(f"This will be printed by process {state.local_process_index}") ``` N)rxrtrgrrrlocal_main_process_firstsz%PartialState.local_main_process_firstNzCallable[..., Any])functioncCs"|jstd|js|js|StS)a Decorator that only runs the decorated function on the main process. Args: function (`Callable`): The function to decorate. Example: ```python >>> from accelerate.state import PartialState >>> state = PartialState() >>> @state.on_main_process ... def print_something(): ... print("This will be printed by process 0 only.") >>> print_something() "This will be printed by process 0 only" ``` zUThe `PartialState` or `Accelerator` must be initialized before calling this function.)rAr]rrrmr$r(rrrron_main_processs  zPartialState.on_main_processcCs|js |js|StS)a Decorator that only runs the decorated function on the local main process. Args: function (`Callable`): The function to decorate. Example: ```python # Assume we have 2 servers with 4 processes each. from accelerate.state import PartialState state = PartialState() @state.on_local_main_process def print_something(): print("This will be printed by process 0 only on each server.") print_something() # On server 1: "This will be printed by process 0 only" # On server 2: "This will be printed by process 0 only" ``` )rtrmr$rrrron_local_main_processs z"PartialState.on_local_main_processcCs|js |js|StS)a Decorator that only runs the decorated function on the last process. Args: function (`Callable`): The function to decorate. Example: ```python # Assume we have 4 processes. from accelerate.state import PartialState state = PartialState() @state.on_last_process def print_something(): print(f"Printed on process {state.process_index}") print_something() "Printed on process 3" ``` )rormr$rrrron_last_process;s zPartialState.on_last_processrO)rrNcCs.|dkrt|j|dS|j|ks&|js*|StS)a Decorator that only runs the decorated function on the process with the given index. Args: function (`Callable`, `optional`): The function to decorate. process_index (`int`, `optional`): The index of the process on which to run the function. Example: ```python # Assume we have 4 processes. from accelerate.state import PartialState state = PartialState() @state.on_process(process_index=2) def print_something(): print(f"Printed on process {state.process_index}") print_something() "Printed on process 2" ``` N)rN)r on_processrNrmr$)r(rrNrrrrWs zPartialState.on_process)rrPcCs.|dkrt|j|dS|j|ks&|js*|StS)aO Decorator that only runs the decorated function on the process with the given index on the current node. Args: function (`Callable`, *optional*): The function to decorate. local_process_index (`int`, *optional*): The index of the local process on which to run the function. Example: ```python # Assume we have 2 servers with 4 processes each. from accelerate import Accelerator accelerator = Accelerator() @accelerator.on_local_process(local_process_index=2) def print_something(): print(f"Printed on process {accelerator.local_process_index}") print_something() # On server 1: "Printed on process 2" # On server 2: "Printed on process 2" ``` N)rP)ron_local_processrPrmr$)r(rrPrrrrxs zPartialState.on_local_processcOs|jrt||dSr!)rtprintr(r"r#rrrrszPartialState.printz torch.devicecCs\trdtjd<tdStjr.tdStr>tdStrNtdStdSdS) a  Returns the default device which is: - MPS if `torch.backends.mps.is_available()` and `torch.backends.mps.is_built()` both return True. - CUDA if `torch.cuda.is_available()` - NPU if `is_npu_available()` - CPU otherwise 1ZPYTORCH_ENABLE_MPS_FALLBACKZmpsr9zxpu:0r>r4N) rrBrCrErFr9rVrrrgrrrrbs       zPartialState.default_device)F)F)N)N)NN)NN)r/r0r1r2 SharedDictrr)rh staticmethodrkpropertyrArmrorrrtrvrxrrrrrrrrrrrbrrrrr3dsB0   " N  !$r3c@seZdZdZeZd+ddddddZedd d d Zd d Z d,ddZ eddZ eddZ e d-ddddZeddZedd ddZedd ddZedd ddZdd Zed.d!dd"d#d$Zed%d&Zed'd(Zd)d*ZdS)/ra Singleton class that has information about the current training environment. **Available attributes:** - **device** (`torch.device`) -- The device to use. - **distributed_type** ([`~accelerate.state.DistributedType`]) -- The type of distributed environment currently in use. - **initialized** (`bool`) -- Whether or not the `AcceleratorState` has been initialized from `Accelerator`. - **local_process_index** (`int`) -- The index of the current process on the current server. - **mixed_precision** (`str`) -- Whether or not the current script will use mixed precision, and if so the type of mixed precision being performed. (Choose from 'no','fp16','bf16 or 'fp8'). - **num_processes** (`int`) -- The number of processes currently launched in parallel. - **process_index** (`int`) -- The index of the current process. - **is_last_process** (`bool`) -- Whether or not the current process is the last one. - **is_main_process** (`bool`) -- Whether or not the current process is the main one. - **is_local_main_process** (`bool`) -- Whether or not the current process is the main one on the local node. - **debug** (`bool`) -- Whether or not the current script is being run in debug mode. NFr\r)mixed_precisionr4_from_acceleratorc Ks|j|_tdrd}tjikr*t|f||jtj||||jsd|_d|_|dkrjt ddn| }|dkrt st d||_ |st d|jtjkrdn||_|jtjkr|dkrtjd rtd tjd <td tjd <d|_n"td tjd <td tjd <d|_ndtjdddkrB|sB||_n@|jtjkrtjdddkrtj|_|jdkr||j||_tjdddkrtj|_||j||_n|jtjkrtjdddkrtj|_|jdkr||j||_n~|jtjtj tj!fkrt"r4tddd|_nd|_|jtj krtjdddkrtj|_|jdkr|||j||_|j j#t$j!kr|jdkr|j%j&dkrdt'j(j)j*_+|jtjd<dS)NZACCELERATE_USE_CPUTZACCELERATE_MIXED_PRECISIONr=Zfp8zDUsing `fp8` precision requires `transformer_engine` to be installed.zPlease make sure to properly initialize your accelerator via `accelerator = Accelerator()` before using any functionality from the `accelerate` library.bf16ZACCELERATE_DOWNCAST_BF16rZ XLA_USE_BF16rZXLA_DOWNCAST_BF16Fr:r5r6ZACCELERATE_USE_FSDPZACCELERATE_USE_MEGATRON_LMZACCELERATE_USE_IPEX)defaultr9rK),rr@rr3update_check_initializedrAdeepspeed_pluginZuse_ipexrlowerrr] dynamo_pluginrKr rTrUrQrBrCrDr\Zdowncast_bfloatrJruZset_mixed_precision fsdp_pluginrqmegatron_lm_pluginrWrYrXrIrr7r rFtyperEbackendsr9matmulZ allow_tf32) r(rr4rrrrrr#rrrr)s                   zAcceleratorState.__init__rcCs |jtjkSr!)rr3rgrrrrA"szAcceleratorState.initializedcCs<td|jd}|jtjkr8|d|jjd7}|S)Nz Mixed precision type: rfz ds_config: )r3rhrrKr rTrdeepspeed_config)r(reprrrrrh&s zAcceleratorState.__repr__cCsd|jr`d}|r*|jjdkr*t|jdd|dk r`||jkr`|jtjkr`t|jd|dddS)zeChecks if a modification is trying to be made and the `AcceleratorState` has already been initializedzAcceleratorState has already been initialized and cannot be changed, restart your runtime completely and pass `{flag}` to `Accelerator()`.r4zcpu=True)flagNzmixed_precision='') rArFrr]formatrUrKr rT)r(rr4errrrrr,s z#AcceleratorState._check_initializedcCstdt|jdkS)NzThe `use_fp16` property is deprecated and will be removed in version 1.0 of Accelerate use `AcceleratorState.mixed_precision == 'fp16'` instead.r=)r`ra FutureWarningrUrgrrruse_fp16:s zAcceleratorState.use_fp16cCsX|jtjkrN|jj}|diddr.d}qT|diddrHd}qTd}n|j}|S)NZfp16enabledFrr=)rKr rTrrrDrU)r(configrrrrrCs z AcceleratorState.mixed_precisionZreset_partial_statecCstj|rtdSri)rrrjr3rkrrrrrkQs zAcceleratorState._reset_statecCstjS)rl)r3rmrgrrrrmXsz AcceleratorState.use_distributedcCstjS)rn)r3rorgrrrro_sz AcceleratorState.is_last_processcCstjS)rp)r3rrrgrrrrrdsz AcceleratorState.is_main_processcCstjS)rs)r3rtrgrrrrtisz&AcceleratorState.is_local_main_processcCstdSr!)r3rvrgrrrrvnsz"AcceleratorState.wait_for_everyoneryrzc cs&tj||d }|VW5QRXdS)a Splits `input` between `self.num_processes` quickly and can be then used on that process. Useful when doing distributed inference, such as with different prompts. Note that when using a `dict`, all keys need to have the same number of elements. Args: inputs (`list`, `tuple`, `torch.Tensor`, or `dict` of `list`/`tuple`/`torch.Tensor`): The input to split between processes. apply_padding (`bool`, `optional`, defaults to `False`): Whether to apply padding by repeating the last element of the input so that all processes have the same number of elements. Useful when trying to perform actions such as `gather()` on the outputs or passing in less inputs than there are processes. If so, just remember to drop the padded elements afterwards. Example: ```python # Assume there are two processes from accelerate.state import AcceleratorState state = AcceleratorState() with state.split_between_processes(["A", "B", "C"]) as inputs: print(inputs) # Process 0 ["A", "B"] # Process 1 ["C"] with state.split_between_processes(["A", "B", "C"], apply_padding=True) as inputs: print(inputs) # Process 0 ["A", "B"] # Process 1 ["C", "C"] ``` )r|N)r3r)r(r{r|rrrrqs'z(AcceleratorState.split_between_processesc cs t dVW5QRXdS)z Lets the main process go first inside a with block. The other processes will enter the with block after the main process exits. N)r3rrgrrrrs z#AcceleratorState.main_process_firstc cs t dVW5QRXdS)z Lets the local main process go inside a with block. The other processes will enter the with block after the main process exits. N)r3rrgrrrrs z)AcceleratorState.local_main_process_firstcOstj||dSr!)r3rrrrrrszAcceleratorState.print)NFNNNNF)NN)F)F)r/r0r1r2rrr)rrArhrrrrrkrmrorrrtrvrrrrrrrrrrsHU    )  rc@seZdZdZeZd"ddddZeddd d Zed dd d Z ed dddZ ed dddZ ed dddZ eddddZ ddZddZddZddZed dddZed d!ZdS)# GradientStateaE Singleton class that has information related to gradient synchronization for gradient accumulation **Available attributes:** - **end_of_dataloader** (`bool`) -- Whether we have reached the end the current dataloader - **remainder** (`int`) -- The number of extra samples that were added from padding the dataloader - **sync_gradients** (`bool`) -- Whether the gradients should be synced across all devices - **active_dataloader** (`Optional[DataLoader]`) -- The dataloader that is currently being iterated over - **dataloader_references** (`List[Optional[DataLoader]]`) -- A list of references to the dataloaders that are being iterated over - **num_steps** (`int`) -- The number of steps to accumulate over - **adjust_scheduler** (`bool`) -- Whether the scheduler should be adjusted to account for the gradient accumulation - **sync_with_dataloader** (`bool`) -- Whether the gradients should be synced at the end of the dataloader iteration and the number of total steps reset Nz$Optional[GradientAccumulationPlugin])gradient_accumulation_plugincCs\|j|_|js8d|_d|_dg|_|dk r2|ni|_|dk rX|j|krX||_dS)NT)rr@rAsync_gradientsactive_dataloaderdataloader_referencesZ to_kwargs plugin_kwargs)r(rrrrr)szGradientState.__init__rOrcCs|jddS)z.Returns the number of steps to accumulate over num_stepsrrrDrgrrrrszGradientState.num_stepsrcCs|jddS)z0Returns whether the scheduler should be adjustedadjust_schedulerFrrgrrrrszGradientState.adjust_schedulercCs|jddS)zyReturns whether the gradients should be synced at the end of the dataloader iteration and the number of total steps resetsync_with_dataloaderTrrgrrrrsz"GradientState.sync_with_dataloadercCs tjikS)z8Returns whether the `GradientState` has been initialized)rrrgrrrrAszGradientState.initializedcCs|js dS|jjS)zAReturns whether we have reached the end of the current dataloaderF) in_dataloaderrend_of_dataloaderrgrrrrszGradientState.end_of_dataloadercCs|js dS|jjS)zOReturns the number of extra samples that were added from padding the dataloaderr8)rr remainderrgrrrrszGradientState.remainderc Cs&d|jd|jd|jd|jd S)NzSync Gradients: z At end of current dataloader: z Extra samples added: z Gradient accumulation plugin: rf)rrrrrgrrrrhs$zGradientState.__repr__cCs ||_dS)zhPrivate function that sets whether gradients should be synchronized. Users should not have to call this.N)r)r(rrrr_set_sync_gradientssz!GradientState._set_sync_gradientscCs||_|j|jdS)zPrivate function that adds a dataloader to `self.dataloader_references` and sets `in_dataloader` to `True`. Users should not have to call this.N)rrappendr(Z dataloaderrrr_add_dataloaderszGradientState._add_dataloadercCs|j||jd|_dS)zPrivate function that removes a dataloader from `self.dataloader_references` and sets `in_dataloader` to `False` if there are no more dataloaders. Users should not have to call this.r8N)rremoverrrrr_remove_dataloader s z GradientState._remove_dataloadercCs |jdk S)z6Returns whether the current process is in a dataloaderN)rrgrrrrszGradientState.in_dataloadercCstjdSri)rrrjrrrrrkszGradientState._reset_state)N)r/r0r1r2rrr)rrrrrArrrhrrrrrrkrrrrrs.r)/ __future__rrrB threadingr` contextlibr functoolsrtypingrrrrEutilsr r r r r rrrrrrrrrrZutils.dataclassesrZtorch_xla.core.xla_modelcoreZ xla_modelrRZ torch_npur r$localr%rrr3rrrrrrs4   D    V~