U
0-eu( @ sh d dl Z d dlZd dlZd dlZddlmZmZ ddlmZm Z m
Z
m Z dd Z dd d Z
dddZdS ) N )AcceleratorState PartialState)
PrecisionTypePrepareForLaunchis_mps_availablepatch_environmentc C s
t } dS )z+Verify a `PartialState` can be initialized.N)r )_ r
U/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/accelerate/launchers.py test_launch s r r
no29500 127.0.0.1c C s& d}d} t dd tj D r&d} n dtjkrFdttjd k}zt| }W n4 t
k
r t
d|j dt d Y nX |s| rtj
d
d d k rd d lm m}
ttjd krt
d
|d krd}t| dd} td| d |
j| ||dd n|r@tj r,td ntd | | n|d krRt
d||krdt
d|dkrd dlm} d dl m }
ttjd krt
d t |||| |||d tj
d d d krRtt d!d} z| | d"|dd W n^ |
k
rP } z>d#}d$|j d kr,t!| d%|nt!| d&| |W 5 d }~X Y nX t| d!d} td'| d( z| | ||dd W nP |
k
r } z0d$|j d krt!d)|nt!d*| |W 5 d }~X Y nX W 5 Q R X nBt" rd+tjd,< td- n tj rtd ntd. | | d S )/a
Launches a training function, using several processes or multiple nodes if it's possible in the current environment
(TPU with multiple cores for instance).
To use this function absolutely zero calls to a CUDA device must be made in the notebook session before calling. If
any have been made, you will need to restart the notebook and make sure no cells use any CUDA capability.
Setting `ACCELERATE_DEBUG_MODE="1"` in your environment will run a test before truly launching to ensure that none
of those calls have been made.
Args:
function (`Callable`):
The training function to execute. If it accepts arguments, the first argument should be the index of the
process run.
args (`Tuple`):
Tuple of arguments to pass to the function (it will receive `*args`).
num_processes (`int`, *optional*):
The number of processes to use for training. Will default to 8 in Colab/Kaggle if a TPU is available, to
the number of GPUs available otherwise.
mixed_precision (`str`, *optional*, defaults to `"no"`):
If `fp16` or `bf16`, will use mixed precision training on multi-GPU.
use_port (`str`, *optional*, defaults to `"29500"`):
The port to use to communicate between processes when launching a multi-GPU training.
master_addr (`str`, *optional*, defaults to `"127.0.0.1"`):
The address to use for communication between processes.
node_rank (`int`, *optional*, defaults to 0):
The rank of the current node.
num_nodes (`int`, *optional*, defaults to 1):
The number of nodes to use for training.
Example:
```python
# Assume this is defined in a Jupyter Notebook on an instance with two GPUs
from accelerate import notebook_launcher
def train(*args):
# Your training function here
...
notebook_launcher(train, args=(arg1, arg2), num_processes=2, mixed_precision="fp16")
```
Fc s s | ]}| d V qdS )ZKAGGLEN)
startswith).0keyr
r
r \ s z$notebook_launcher..TZIPythonz google.colabz Unknown mixed_precision mode: z. Choose between .ZTPU_NAMENr zTo train on TPU in Colab or Kaggle Kernel, the `Accelerator` should only be initialized inside your training function. Restart your notebook and make sure no cells initializes an `Accelerator`. ZTPU)Zdistributed_typezLaunching a training on z TPU cores.forkargsZnprocsZ start_methodz Launching training on one GPU.z Launching training on one CPU.zcYou have to specify the number of GPUs you would like to use, add `num_processes=...` to your call.z4The node_rank must be less than the number of nodes.r start_processes)ProcessRaisedExceptionzTo launch a multi-GPU training from your notebook, the `Accelerator` should only be initialized inside your training function. Restart your notebook and make sure no cells initializes an `Accelerator`.)Znproc node_rank
world_size master_addr master_portmixed_precisionZACCELERATE_DEBUG_MODEfalsetrueZ MULTI_GPUr
zQAn issue was found when verifying a stable environment for the notebook launcher.z.Cannot re-initialize CUDA in forked subprocesszThis likely stems from an outside import causing issues once the `notebook_launcher()` is called. Please review your imports and test them when running the `notebook_launcher()` to identify which one is problematic and causing CUDA to be initialized.z! The following error was raised: zLaunching training on z GPUs.aU CUDA has been initialized before the `notebook_launcher` could create a forked subprocess. This likely stems from an outside import causing issues once the `notebook_launcher()` is called. Please review your imports and test them when running the `notebook_launcher()` to identify which one is problematic and causing CUDA to be initialized.z0An issue was found when launching the training: 1ZPYTORCH_ENABLE_MPS_FALLBACKzLaunching training on MPS.zLaunching training on CPU.)#anyosenvironkeyssysmodulesstr get_ipythonr lower
ValueErrorr listgetZ)torch_xla.distributed.xla_multiprocessing distributedZxla_multiprocessinglenr Z
_shared_stater printspawntorchcudaZ is_availabletorch.multiprocessingr Ztorch.multiprocessing.spawnr r r r RuntimeErrorr )functionr
num_processesr Zuse_portr r Z num_nodesZin_colabZ in_kaggleZxmplauncherr r eerrr
r
r notebook_launcher s <
& .
r= c
C sb ddl m} t D}t|ddd|jdd" t| dd }||||d
d W 5 Q R X W 5 Q R X d S )
aC
Launches a training function using several processes on CPU for debugging purposes.
This function is provided for internal testing and debugging, but it's not intended for real trainings. It will
only use the CPU.
Args:
function (`Callable`):
The training function to execute.
args (`Tuple`):
Tuple of arguments to pass to the function (it will receive `*args`).
num_processes (`int`, *optional*, defaults to 2):
The number of processes to use for training.
r r z127.0.01r r
yes)r r r Zaccelerate_mixed_precisionZaccelerate_debug_rdv_fileZaccelerate_use_cpuT)debugr r N)r6 r tempfileNamedTemporaryFiler namer )r8 r r9 r Ztmp_filer: r
r
r debug_launcher s
rD )r
Nr
r r r r )r
r> )r% r( rA r4 stater r utilsr r r r r r= rD r
r
r
r s
*