U 0-et@sddlZddlZddlZddlZddlZddlZddlmZddlZddl Z ddl m Z m Z ddl mZddlmZddlmZddlmZmZmZmZmZmZmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%ddl&m'Z'm(Z(erdd l)m*Z*dd l+m,Z,d Z-ej.e-d e,gd e/e0Z1ddddddZ2ddZ3Gdddej4Z5d.ddZ6ddZ7ddZ8ddZ9d d!Z:d"d#Z;ed$d%d&Zd+d,Z?e0d-kre?dS)/N)Path)default_config_fileload_config_from_file)SageMakerConfig)DYNAMO_BACKENDS)get_int_from_env)ComputeEnvironmentDistributedTypePrepareForLaunch _filter_argsis_bf16_availableis_deepspeed_availableis_npu_availableis_rich_availableis_sagemaker_availableis_torch_versionis_tpu_availableis_xpu_availablepatch_environmentprepare_deepspeed_cmd_envprepare_multi_gpu_envprepare_sagemager_args_inputsprepare_simple_launcher_cmd_env prepare_tpu)DEEPSPEED_MULTINODE_LAUNCHERSTORCH_DYNAMO_MODES) get_console) RichHandlerz %(message)sz[%X])formatdatefmthandlersDistributed GPUsTPUDeepSpeed ArgumentsFSDP ArgumentsMegatron-LM Arguments)z --multi-gpu--tpu--use_deepspeed --use_fsdp--use_megatron_lmcCs.|dr*|dd|ddddSdS)zIFinds all cases of - after the first two characters and changes them to _z--N-_) startswithreplace)optionr0[/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/accelerate/commands/launch.py clean_optionJs r2cs"eZdZdZdfdd ZZS)_CustomHelpActiona  This is a custom help action that will hide all arguments that are not used in the command line when the help is called. This is useful for the case where the user is using a specific platform and only wants to see the arguments for that platform. Nc sdtjdkr0dtjddkr0tjdd}ntjdd}|j}dddd d g}t|dkrd d |D}ttt|}d d |D} t|D]\} } | jj || krt || dt j q| jj dkr t | jt |rt || dt j nt || d| jdq| jj dkrt | jt |rBt || dt j qt || d| jdqtt|jD],\} } tdd | jDrh|j| qht||||dS)NZ acceleraterlaunchHardware Selection ArgumentsResource Selection ArgumentsTraining Paradigm Argumentszpositional argumentszoptional argumentscSsg|]}|tkr|qSr0)options_to_groupkeys.0argr0r0r1 es z._CustomHelpAction.__call__..cSsg|] }t|qSr0)r:)r=or0r0r1r?gshelpz (currently selected)cSsg|]}|jtjkqSr0)rAargparseSUPPRESSr<r0r0r1r?zs)sysargv_actionslenlistmapr2 enumerate containertitlesetattrrBrCsetoption_strings isdisjointrA_action_groupsall_group_actionsremovesuper__call__) selfparser namespacevalues option_stringargsoptstitlesZused_platformsZ used_titlesir>group __class__r0r1rVWs:  z_CustomHelpAction.__call__)N)__name__ __module__ __qualname____doc__rV __classcell__r0r0rar1r3Psr3c Cs|dk r|jdddd}ntjdddd}|ddt|jdddd d |jd dd d |jddddd |dd}|jddddd|jddddd|jddddd|jddddd|dd}|jdtd d!d"d#gd$d%|jd&tdd'd(|jd)tdd*d(|jd+tdd,d(|jd-td gd.d/t Dd0d%|jd1td2t d3d4|jd5ddd6d|jd7ddd8d|d9d:}|jd;dddd|jd?ddd@d|jdAdddBd|dCdD}|jdEddFd |jdGdddHd|jdItddJd(|jdKtddLd(|jdMtddNd(|jdOdPdQtdRdS|jdTtd2dUd(|jdVtdWdXd(|jdYtdZd[d(|jd\td]d^d(|jd_t d`dad(|jdbdcdddd |jdeddfd |dgdh}|jdiddjdkdl|jdmdndjdodl|jdpddqd |jdrtdsdtdu|jdvtdsdwdu|jdxtddyd(|jdzdd{d |d|d}}|jd~dtddS|jddtddS|jddtddS|jddtddS|jddtddS|jddtddS|jddtddS|jddt ddS|jddtddS|jddtddS|jddtddS|jddtddS|jddtddS|jddtddS|dd}|jddtddS|jdtddd(|jdtddd(|jdtddd(|jddtddS|jddtddS|jddtddS|jddtddS|jddtddS|jddtddS|dd} | jdtddd(| jdtddd(| jdtddd(| jddtddS| jddtddS| jddtddS| jddt ddS|ddơ} | jdtddd(| jdtddd(|jdddd |jdtddύ|jdtj ddҍ|dk r|j tdӍ|S)Nr4F)add_help allow_abbrevzAccelerate launch commandactionrAz-hz--helpz Show this help message and exit.)rjrAz --config_filezFThe config file to use for the default values in the launching script.)defaultrAz--quietz-q store_truezSilence subprocess errors from the launch stack trace and only show the relevant tracebacks. (Only applicable to DeepSpeed and single-process configurations)r7z0Arguments for selecting the hardware to be used.z--cpuz0Whether or not to force the training on the CPU.)rkrjrAz --multi_gpuz=Whether or not this should launch a distributed GPU training.r&z1Whether or not this should launch a TPU training.z--ipexzLWhether or not this should launch a Intel PyTorch Extension (IPEX) training.r8z@Arguments for fine-tuning how available hardware should be used.--mixed_precisionnoZfp16bf16Zfp8zWhether or not to use mixed precision training. Choose between FP16 and BF16 (bfloat16) training. BF16 training is only supported on Nvidia Ampere GPUs and PyTorch 1.10 or later.)typechoicesrA--num_processesz9The total number of processes to be launched in parallel.)rprkrA--num_machinesz3The total number of machines used in this training.z--num_cpu_threads_per_processzLThe number of CPU threads per process. Can be tuned for optimal performance.--dynamo_backendcSsg|] }|qSr0)lower)r=br0r0r1r?sz)launch_command_parser..zkChoose a backend to optimize your training with dynamo, see more at https://github.com/pytorch/torchdynamo.z --dynamo_moderkz4Choose a mode to optimize your training with dynamo.)rprkrqrAz--dynamo_use_fullgraphz[Whether to use full graph mode for dynamo or it is ok to break model into several subgraphsz--dynamo_use_dynamicz(Whether to enable dynamic shape tracing.r9z;Arguments for selecting which training paradigm to be used.r'zWhether to use deepspeed.r(zWhether to use fsdp.r)zWhether to use Megatron-LM.z --use_xpuzDWhether to use IPEX plugin to speed up training on XPU specifically.r!z.Arguments related to distributed GPU training.z --gpu_idszWWhat GPUs (by id) should be used for training on this machine as a comma-seperated listz--same_networkzQWhether all machines used for multinode training exist on the same local network.z--machine_rankz9The rank of the machine on which this script is launched.z--main_process_ipz(The IP address of the machine of rank 0.z--main_process_portz:The port to use to communicate with the machine of rank 0.z-tz--tee0z4Tee std streams into a log file and also to console.)rkrprAz--rolez"User-defined role for the workers.z--rdzv_backendZstaticzFThe rendezvous method to use, such as 'static' (the default) or 'c10d'z --rdzv_confzJAdditional rendezvous configuration (=,=,...).z--max_restartsrz7Maximum number of worker group restarts before failing.z--monitor_intervalz6Interval, in seconds, to monitor the state of workers.z-mz--modulezwChange each process to interpret the launch script as a Python module, executing with the same behavior as 'python -m'.z --no_pythonz|Skip prepending the training script with 'python' - just execute it directly. Useful when the script is not a Python script.r"zArguments related to TPU.z --tpu_clustertpu_use_clusterz*Whether to use a GCP TPU pod for training.)rjdestrA--no_tpu_cluster store_falsez?Should not be passed explicitly, this is for internal use only.z--tpu_use_sudozGWhether to use `sudo` when running the TPU training script in each pod.z--vmappendzkList of single Compute VM instance names. If not provided we assume usage of instance groups. For TPU pods.)rprjrAz--envzOList of environment variables to set on the Compute VM instances. For TPU pods.--main_training_functionzTThe name of the main function to be executed in your script (only for TPU training).z--downcast_bf16zWhether when using bf16 precision on TPUs if both float and double tensors are cast to bfloat16 or if double tensors remain as float32.r#zArguments related to DeepSpeed.z--deepspeed_config_filezDeepSpeed config file.z --zero_stagez{DeepSpeed's ZeRO optimization stage (useful only when `use_deepspeed` flag is passed). If unspecified, will default to `2`.z--offload_optimizer_devicezDecides where (none|cpu|nvme) to offload optimizer states (useful only when `use_deepspeed` flag is passed). If unspecified, will default to 'none'.z--offload_param_devicezDecides where (none|cpu|nvme) to offload parameters (useful only when `use_deepspeed` flag is passed). If unspecified, will default to 'none'.z--offload_optimizer_nvme_pathzDecides Nvme Path to offload optimizer states (useful only when `use_deepspeed` flag is passed). If unspecified, will default to 'none'.z--offload_param_nvme_pathzDecides Nvme Path to offload parameters (useful only when `use_deepspeed` flag is passed). If unspecified, will default to 'none'.z--gradient_accumulation_stepszNo of gradient_accumulation_steps used in your training script (useful only when `use_deepspeed` flag is passed). If unspecified, will default to `1`.z--gradient_clippingzgradient clipping value used in your training script (useful only when `use_deepspeed` flag is passed). If unspecified, will default to `1.0`.z--zero3_init_flagzDecides Whether (true|false) to enable `deepspeed.zero.Init` for constructing massive models. Only applicable with DeepSpeed ZeRO Stage-3. If unspecified, will default to `true`.z--zero3_save_16bit_modelzDecides Whether (true|false) to save 16-bit model weights when using ZeRO Stage-3. Only applicable with DeepSpeed ZeRO Stage-3. If unspecified, will default to `false`.z--deepspeed_hostfilez@DeepSpeed hostfile for configuring multi-node compute resources.z--deepspeed_exclusion_filterz>DeepSpeed exclusion filter string when using mutli-node setup.z--deepspeed_inclusion_filterz>DeepSpeed inclusion filter string when using mutli-node setup.z--deepspeed_multinode_launcherzMDeepSpeed multi-node launcher to use. If unspecified, will default to `pdsh`.r$z3Arguments related to Fully Shared Data Parallelism.z--fsdp_offload_paramsfalsezvDecides Whether (true|false) to offload parameters and gradients to CPU. (useful only when `use_fsdp` flag is passed).z--fsdp_min_num_paramsgחAzlFSDP's minimum number of parameters for Default Auto Wrapping. (useful only when `use_fsdp` flag is passed).z--fsdp_sharding_strategyr5zGFSDP's Sharding Strategy. (useful only when `use_fsdp` flag is passed).z--fsdp_auto_wrap_policyzFFSDP's auto wrap policy. (useful only when `use_fsdp` flag is passed).z$--fsdp_transformer_layer_cls_to_wrapzTransformer layer class name (case-sensitive) to wrap ,e.g, `BertLayer`, `GPTJBlock`, `T5Block` .... (useful only when `use_fsdp` flag is passed).z--fsdp_backward_prefetch_policyzNFSDP's backward prefetch policy. (useful only when `use_fsdp` flag is passed).z--fsdp_state_dict_typezEFSDP's state dict type. (useful only when `use_fsdp` flag is passed).z--fsdp_forward_prefetchzIf True, then FSDP explicitly prefetches the next upcoming all-gather while executing in the forward pass (useful only when `use_fsdp` flag is passed).z--fsdp_use_orig_paramszIf True, allows non-uniform `requires_grad` during init, which means support for interspersed frozen and trainable paramteres. (useful only when `use_fsdp` flag is passed).z--fsdp_sync_module_statestruezIf True, each individually wrapped FSDP unit will broadcast module parameters from rank 0. (useful only when `use_fsdp` flag is passed).r%z!Arguments related to Megatron-LM.z--megatron_lm_tp_degreezbMegatron-LM's Tensor Parallelism (TP) degree. (useful only when `use_megatron_lm` flag is passed).z--megatron_lm_pp_degreezdMegatron-LM's Pipeline Parallelism (PP) degree. (useful only when `use_megatron_lm` flag is passed).z--megatron_lm_num_micro_batchesznMegatron-LM's number of micro batches when PP degree > 1. (useful only when `use_megatron_lm` flag is passed).z"--megatron_lm_sequence_parallelismzDecides Whether (true|false) to enable Sequence Parallelism when TP degree > 1. (useful only when `use_megatron_lm` flag is passed).z#--megatron_lm_recompute_activationszDecides Whether (true|false) to enable Selective Activation Recomputation. (useful only when `use_megatron_lm` flag is passed).z'--megatron_lm_use_distributed_optimizerzDecides Whether (true|false) to use distributed optimizer which shards optimizer state and gradients across Data Pralellel (DP) ranks. (useful only when `use_megatron_lm` flag is passed).z--megatron_lm_gradient_clippingg?zMegatron-LM's gradient clipping value based on global L2 Norm (0 to disable). (useful only when `use_megatron_lm` flag is passed).z AWS ArgumentszArguments related to AWS.z--aws_access_key_idzFThe AWS_ACCESS_KEY_ID used to launch the Amazon SageMaker training jobz--aws_secret_access_keyzKThe AWS_SECRET_ACCESS_KEY used to launch the Amazon SageMaker training job.z--debugzLWhether to print out the torch.distributed stack trace when something fails.training_scriptznThe full path to the script to be launched in parallel, followed by all the arguments for the training script.)rprAtraining_script_argsz!Arguments of the training script.)nargsrA)func) add_parserrBArgumentParserregisterr3 add_argumentadd_argument_groupstrintrrfloat REMAINDER set_defaultslaunch_command) Z subparsersrXZ hardware_argsZ resource_argsZ paradigm_argsZdistributed_argsZtpu_argsZdeepspeed_argsZ fsdp_argsZmegatron_lm_argsZaws_argsr0r0r1launch_command_parsers           rcCsRt|\}}tj||d}||jdkrN|jsDtj|j|dn tddS)Nenvr returncodecmdr5) r subprocessPopenwaitrquietCalledProcessErrorrDexit)r\r current_envprocessr0r0r1simple_launcherms  rc Csddlmm}t|}t|dd}t||d|jd|jg}t f|Xz||WnBt k rt r|rt }| d|jtgddnYnXW5QRXdS)NrdebugF--training_script--training_script_argsE [bold red]Using --debug, `torch.distributed` Stack Trace:[/bold red]suppress show_locals)torch.distributed.run distributedrunrgetattrr get_args_parserrrr Exceptionrrprintprint_exception__file__)r\ distrib_runrrconsoler0r0r1multi_gpu_launcherys"    rc Csrddlmm}ts tdt|\}}|jdkr|jtdkrt dd@}| D]0\}}d|ksXd|krrqX| |d|d qXW5QRXt j ||d }||jdkr|jst j|j|d n tdnt|d d }t||d|jd|jg}tf|^z||WnHtk rbtr\|r\t} | d| jtgd dnYnXW5QRXdS)NrzSDeepSpeed is not installed => run `pip3 install deepspeed` or build it from source.r5z.deepspeed_enva; = rrrFrrrr) rrrr ImportErrorr num_machinesZdeepspeed_multinode_launcherropenitemswriterrrrrrrDrrr rrrrrrrrrr) r\rrrfkeyvaluerrrr0r0r1deepspeed_launchers>  "     rc Csddlmm}|jr tdt|i\}}|jr<|j}n&t|j}t j t |j |j}t|}t||jstd|jd|jg|jt _t||j}tf||jt|d|jdW5QRXdS)Nrz,--no_python cannot be used with TPU launcherz2Your training script should have a function named zE, or you should pass a different value to `--main_training_function`.r0)r\Znprocs)Z)torch_xla.distributed.xla_multiprocessingrZxla_multiprocessing no_python ValueErrorrmodulerrrDpathr~rparentresolvestem importlib import_modulehasattrmain_training_functionrrrErrspawnr num_processes)r\Zxmprmod_nameZ script_pathmodZ main_functionr0r0r1 tpu_launchers$      rc Csddlm}i}t||d\}}t|dd}|j}|j}t||d|jddd g}|j rbd g}ng}|d dd d t ddddddt |j dt |j |g|7}||_ d}t|D]<} | drt|| } | dkr| dk r|| d| d7}q|dkrtd|dd|D|_|jdz||WnHtk r|trv|rvt} | d| jtgddnYnXdS)Nr)xla_distTrFr&z --positionalrxz--restart-tpuvm-pod-serverZsudozaccelerate-launchr|rsr5rmrnrtrrrZdocker_z="z" zgDocker containers are not supported for TPU pod launcher currently, please remove the following flags: cSsg|]\}}|d|qS)rr0)r=kvr0r0r1r?sz$tpu_pod_launcher..zACCELERATE_IN_TPU_POD=1zF [bold red]Using --debug, `torch_xla.xla_dist` Stack Trace:[/bold red]r)Ztorch_xla.distributedrrrrrr rZtpu_nameZ tpu_use_sudorrr positionalvarsr-rrrr~Zresolve_and_executerrrrrr) r\rrrrrnew_argsZnew_cmdZ bad_flagsr>rrr0r0r1tpu_pod_launchersj        r)sagemaker_configcCsftstd|js|jr"tdddlm}t||\}}|f|}|j|dt d|j dS)NzsPlease install sagemaker to be able to launch training on Amazon SageMaker with `pip install accelerate[sagemaker]`z`SageMaker requires a python training script file and cannot be used with --module or --no_pythonr) HuggingFace)inputsz!You can find your model data at: ) rrrrrZsagemaker.huggingfacerrfitrZ model_data)rr\rZsagemaker_inputsZhuggingface_estimatorr0r0r1sagemaker_launchers    rcCst|j|j|j|j|jgdkr(td|jrJ|jdk rJ|jdkrJtdd}g}d}|jdk svt j t rl|jslt |j}|js|js|js|js|js|js|jtjk|_|jtjtjtjfkrdnd|_|jtjk|_|jtjk|_|jtjk|_|jr|jnd|_|jdkrD|jdk r>|j|_nd|_|jr`|jdkr`|j|_t|jddkr|jdkr|jr|jdkrtd |jtjkr|j !D]\}}t"|t#r|j$D]}t%|||j$|q|j&D].}|}d |kr d |}t%|||j&|q|j'D]}t%|||j'|q&|j(D]}t%|||j(|qF|j)D]}t%|||j)|qfq|d krt*||ddkrt%|||q|j+s|j+|_+|j,s|j,dkrd |_,n |j,|_,d}nld}d} |j-s|j.rt/j01rt2dd}nt3d}|j,dkrV|sV|jrDt4sVt| j5ddd|j6dkrd |_6n:|jdkr|j.rt7rt/j08|_n"t9rt/j:8|_n t/j;8|_||j.rt7rt/j08dks.t9rt/j:8dks.t/j;8dkr>|+d|dkp|dk o|jtj?k} | rL|j@dkrLd|_@|j-rL|jdkrLtAdddgd} tBtCjDdd| } | dkrL| |_@|G| |||fS)%Nr5zcYou can only use one of `--cpu`, `--multi_gpu`, `--tpu`, `--use_deepspeed`, `--use_fsdp` at a time.r6z:You need to use at least 2 processes to use `--multi_gpu`.FTrR,zLess than two GPU ids were configured and tried to run on on multiple GPUs. Please ensure at least two are specified for `--gpu_ids`, or use `--gpu_ids='all'`.ZfsdpZfsdp_)compute_environmentmixed_precisiondistributed_typernz-{mode} mixed precision requires {requirement}z>=z1.10roz'PyTorch >= 1.10 and a supported device.)mode requirementz* `--num_processes` was set to a value of ``zx More than one GPU was found, enabling multi-GPU training. If this was unintended please pass in `--num_processes=1`.z+ `--num_machines` was set to a value of `1`z1 `--mixed_precision` was set to a value of `'no'`use_cpuz0 `--dynamo_backend` was set to a value of `'no'`zRRunning script in debug mode, expect distributed operations to be slightly slower.ZMPI_LOCALNRANKSZOMPI_COMM_WORLD_LOCAL_SIZEZMV2_COMM_WORLD_LOCAL_SIZE)Zlogicalz- `--num_cpu_threads_per_process` was set to `z9` to improve out-of-box performance when training on CPUsz[The following values were not passed to `accelerate launch` and had defaults used instead: rzh To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.)Hsum multi_gpucputpu use_deepspeeduse_fsdprr config_fileosrisfilerrrzuse_megatron_lmrr Z DEEPSPEEDZ MULTI_GPUZ MULTI_NPUZ MULTI_XPUr"ZFSDPZ MEGATRON_LMZgpu_idsrrGsplitrrZ LOCAL_MACHINE__dict__r isinstancedictdeepspeed_configrMZ fsdp_configZmegatron_lm_configZ dynamo_configZ ipex_configrrrrZuse_xputorchZxpuZ is_availablerr rrZdynamo_backendrZ device_countrZnpucudar~rloggerAMAZON_SAGEMAKERZnum_cpu_threads_per_processrrpsutil cpu_countanyjoinwarning)r\defaultsZwarnedmp_from_config_flagnameattrrZ arg_to_setZ native_amperrZis_aws_env_disabledZ local_sizeZthreads_per_processmessager0r0r1_validate_launch_command's"      6         "                     rcCst|\}}}|jrZ|jsZ|r,t|jng|_|rB|jdd|j|_t |n|j rp|jspt |nz|j r|jst |nd|j r|jst |nN|jr|js|jrt|qt|n(|dk r|jtjkrt||nt|dS)Nrr)rrrrHrr;Z'deepspeed_fields_from_accelerate_configr~rrrrrrrrzrrrrrrr)r\rrr0r0r1rs(             rcCst}|}t|dS)N)r parse_argsr)rXr\r0r0r1mainsr__main__)N)@rBrloggingrrrDpathlibrrrZaccelerate.commands.configrrZ&accelerate.commands.config.config_argsrZ'accelerate.commands.config.config_utilsrZaccelerate.staterZaccelerate.utilsrr r r r r rrrrrrrrrrrrZaccelerate.utils.constantsrrrichrZ rich.loggingrZFORMAT basicConfig getLoggerrcrr:r2 _HelpActionr3rrrrrrrrrrr0r0r0r1sX    P    0 p );