U 0-e\k@sddlZddlmZmZmZmZmZmZmZddl m Z m Z m Z m Z mZmZddlmZddlmZmZmZmZmZmZmZdd ZdS) N)ComputeEnvironmentDistributedTypeis_deepspeed_availableis_mps_availableis_npu_availableis_transformers_availableis_xpu_available)DEEPSPEED_MULTINODE_LAUNCHERSFSDP_AUTO_WRAP_POLICYFSDP_BACKWARD_PREFETCHFSDP_SHARDING_STRATEGYFSDP_STATE_DICT_TYPETORCH_DYNAMO_MODES) ClusterConfig)DYNAMO_BACKENDS _ask_field _ask_options_convert_distributed_mode_convert_dynamo_backend_convert_mixed_precision_convert_yes_no_to_boolc0s tdddddddgt}d}d }d }d}d}d}d }d }d } |tjtjtjtjfkrtd td d}|d krtdt t |t}td}tdt}tdt d dd}|stdd d}tdt d dd} |tj krtdt d dd} n|tjkrd } nd } i} | rtdt d dd| d<| sFt rF|tjtjtjfkrFtdt d dd| d<i} tdt d dd} | rd}tddd tDtd!d| |d"<td#t d dd}|rtd$td%d&dd| |d'<td(t d dd| |d)<td*t d dd| |d+<| ot}i}|tjtj fkr.|s.td,t d dd}|r>tj}ts>td-|tjkr.td.t d dd}|rvtd/td0d|d1<ntd2dd d!d3gtd!d|d4<d0d5d6g|d4d!krtd7fd8d&|d9<td:fd;d&|d<<|d<d6krtd=td>d|d?<|d9d6krtd@td>d|dA<tdBtd d|dC<tdDt d dd}|rTtdEtdFd|dG<|d4d3krvtdHt d dd|dI<tdJt d dd|dK<|dKrtstdL|d kr.dM}t|tdNd&|dO<|dOtd kr.tdPt|dQ<tdRt d dd}|r tdSt|dT<tdUt d dd}|r.tdVt|dW<i}|tjtjtjfkrtdXt d dd}|rdtj}|tjkrdY}t|tdZd&d d|d[<td\t d dd|d]<d^}t|td_d&|d`<|d`tdkrtdat d dd}|stdbt|dc<n$|d`td krtddtded|df<dg}t|tdhd&|di<dj}t|tdkd&d!d|dl<tdmt d dd|dn<tdot d dd|dp<tdqt d dd|dr<i}|tjfkrtdst d dd} | rtj }|tj krdt}tdutd dvd||dw<||dwd krtdxt d dd||dy<tdztd dvd||d{<||d{d krFtd|td dvd||d}<td~t d dd||d<tdt d dd||d<tdtdFd||dG<d}!d}"d}#g}$d}%d}&d}'d }(d })|tjtjtjtjtjfkrt|!dd "dd}*|*dkr|*d7}*n|*d7}*td|*dtd dvd}n,|tjtjtj fkrFtdtd dvd}nd }|tjkrz|d krz|d krzt#d|d|tjtjtjtj fkr| s|st$rd}*nd}*td|*ddd}|tjkrd}+tddd},tdt d dd})|)rtdddd}%tdddd}'tdd dd}(tdt d dd}-|-rtdt d dd}.|.rjtdddd}"t%j&'|"}"n>t(dg}!d }/|/r|!)tddddtdt d dd}/qztddd!d}&tddd!d}$n0d},|tjkr|rd}+ntdddddgt*}+| r|+dk r| st(d|tjk r>|+dk r>tddd}#t+t,j-||||+|#|||||,|||| | |||!|"|$|%|&|'|(|)| | dS)Nz$Which type of machine are you using?zNo distributed trainingz multi-CPUz multi-XPUz multi-GPUz multi-NPUTPUrrZstaticTFzYHow many different machines will you use (use more than 1 for multi-node training)? [1]: )defaultz!What is the rank of this machine?zGWhat is the IP address of the machine that will host the main process? zDWhat is the port you will use to communicate with the main process? zAre all the machines on the same local network? Answer `no` if nodes are on the cloud and/or on different network hosts [YES/no]: zPlease enter yes or no.)r error_messagez?What rendezvous backend will you use? ('static', 'c10d', ...): zShould distributed operations be checked while running for errors? This can avoid timeout issues but will be slower. [yes/NO]: zxDo you want to run your training on CPU only (even if a GPU / Apple Silicon / Ascend NPU device is available)? [yes/NO]:zXDo you want to use Intel PyTorch Extension (IPEX) to speed up training on CPU? [yes/NO]:ZipexzDDo you want to use XPU plugin to speed up training on XPU? [yes/NO]:Zuse_xpuz?Do you wish to optimize your script with torch dynamo?[yes/NO]:Zdynamo_z+Which dynamo backend would you like to use?cSsg|] }|qS)lower).0xrrc/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/accelerate/commands/config/cluster.py sz%get_cluster_input..backendzGDo you want to customize the defaults sent to torch.compile? [yes/NO]: zWhich mode do you want to use?cSs tt|SN)rintrrrr z#get_cluster_input..modez\Do you want the fullgraph mode or it is ok to break model into several subgraphs? [yes/NO]: Z use_fullgraphz7Do you want to enable dynamic shape tracing? [yes/NO]: Z use_dynamicz(Do you want to use DeepSpeed? [yes/NO]: zRDeepSpeed is not installed => run `pip3 install deepspeed` or build it from sourcezDDo you want to specify a json file to a DeepSpeed config? [yes/NO]: z9Please enter the path to the json DeepSpeed config file: noneZdeepspeed_config_filez8What should be your DeepSpeed's ZeRO optimization stage?rZ zero_stagecpuZnvmez"Where to offload optimizer states?cs t|Sr$r%r&Zdeepspeed_devicesrr r'r(Zoffload_optimizer_devicezWhere to offload parameters?cs t|Sr$r,r&r-rr r'r(Zoffload_param_devicez Nvme Path to offload parameters?z/nvmeZoffload_param_nvme_pathz&Nvme Path to offload optimizer states?Zoffload_optimizer_nvme_pathzIHow many gradient accumulation steps you're passing in your script? [1]: Zgradient_accumulation_stepsz0Do you want to use gradient clipping? [yes/NO]: z,What is the gradient clipping value? [1.0]: g?Zgradient_clippingzLDo you want to save 16-bit model weights when using ZeRO Stage-3? [yes/NO]: Zzero3_save_16bit_modelzoDo you want to enable `deepspeed.zero.Init` when using ZeRO Stage-3 for constructing massive models? [yes/NO]: Zzero3_init_flagzpWhen `zero3_init_flag` is set, it requires Transformers to be installed. Please run `pip3 install transformers`.z*Which Type of launcher do you want to use?cSs tt|Sr$)r r%r&rrr r'r(Zdeepspeed_multinode_launcheraEDeepSpeed configures multi-node compute resources with hostfile. Each row is of the format `hostname slots=[num_gpus]`, e.g., `localhost slots=2`; for more information please refer official [documentation](https://www.deepspeed.ai/getting-started/#resource-configuration-multi-node). Please specify the location of hostfile: Zdeepspeed_hostfilez:Do you want to specify exclusion filter string? [yes/NO]: z#DeepSpeed exclusion filter string: Zdeepspeed_exclusion_filterz:Do you want to specify inclusion filter string? [yes/NO]: z#DeepSpeed inclusion filter string: Zdeepspeed_inclusion_filterz7Do you want to use FullyShardedDataParallel? [yes/NO]: z&What should be your sharding strategy?cSs t|dS)Nrr,r&rrr r'Jr(Zfsdp_sharding_strategyzBDo you want to offload parameters and gradients to CPU? [yes/NO]: Zfsdp_offload_paramsz%What should be your auto wrap policy?cSs tt|Sr$)r r%r&rrr r'Wr(Zfsdp_auto_wrap_policyulDo you want to use the model's `_no_split_modules` to wrap. Only applicable for 🤗 Transformers [yes/NO]: zSpecify the comma-separated list of transformer layer class names (case-sensitive) to wrap ,e.g, :`BertLayer`, `GPTJBlock`, `T5Block`, `BertLayer,BertEmbeddings,BertSelfOutput` ...? : Z"fsdp_transformer_layer_cls_to_wrapzaWhat should be your FSDP's minimum number of parameters for Default Auto Wrapping Policy? [1e8]: iZfsdp_min_num_paramsz4What should be your FSDP's backward prefetch policy?cSs tt|Sr$)r r%r&rrr r'pr(Zfsdp_backward_prefetch_policyz+What should be your FSDP's state dict type?cSs tt|Sr$)rr%r&rrr r'vr(Zfsdp_state_dict_typez@Do you want to enable FSDP's forward prefetch policy? [yes/NO]: Zfsdp_forward_prefetchzBDo you want to enable FSDP's `use_orig_params` feature? [yes/NO]: Zfsdp_use_orig_paramszsDo you want each individually wrapped FSDP unit to broadcast module parameters from rank 0 at the start? [YES/no]: Zfsdp_sync_module_statesz+Do you want to use Megatron-LM ? [yes/NO]: Z megatron_lm_z0What is the Tensor Parallelism degree/size? [1]:zPlease enter an integer.Z tp_degreez6Do you want to enable Sequence Parallelism? [YES/no]: Zsequence_parallelismz2What is the Pipeline Parallelism degree/size? [1]:Z pp_degreez)What is the number of micro-batches? [1]:Znum_micro_batcheszDDo you want to enable selective activation recomputation? [YES/no]: Zrecompute_activationsz{Do you want to use distributed optimizer which shards optimizer state and gradients across data pralellel ranks? [YES/no]: Zuse_distributed_optimizerzSWhat is the gradient clipping value based on global L2 Norm (0 to disable)? [1.0]: no.ZMULTI_z coresz(s)z How many z. should be used for distributed training? [1]:z=How many GPU(s) should be used for distributed training? [1]:zSpecified distributed type zy but only using 1 GPU on a single machine. Please select `No distributed training` for the type of machine you are using.zNPU(s)zGPU(s)zWhat zV (by id) should be used for training on this machine as a comma-seperated list? [all]:allziWhat is the name of the function in your script that should be launched in all parallel scripts? [main]: mainz'Are you using a TPU cluster? [yes/NO]: z&What is the name of your TPU cluster? z*Please enter the name of your TPU cluster.z&What is the zone of your TPU cluster? z*Please enter the zone of your TPU cluster.zFTo run a python script in a TPU pod, should `sudo` be used? [yes/NO]: zCDo you have code you wish to run on startup in each pod? [yes/NO]: z1Is this code located in a bash script? [yes/NO]: z&What is the path to your bash script? z*Please enter the path to your bash script.zLPlease enter each command seperately you wish to run on startup in each pod.z(Please enter a single command to be ran zTPlease enter the commands you wish to run on startup in each pod as a single string.z.Do you wish to add another command? [yes/NO]: zqIf not using an instance group, what are the names of the Compute VM instances to be used, seperated by a comma: ,zQWhat environment variables do you wish to set in each pod, seperated by a comma: z2Do you wish to use FP16 or BF16 (mixed precision)?Zfp16Zbf16Zfp8zTorch dynamo used without mixed precision requires TF32 to be efficient. Accelerate will enable it by default when launching your scripts.zWShould `torch.float` be cast as `bfloat16` and `torch.double` remain `float32` on TPUs?)Zcompute_environmentdistributed_type num_processesgpu_idsmixed_precisionZ downcast_bf16 machine_rank num_machinesmain_process_ipmain_process_portmain_training_functiondeepspeed_config fsdp_configmegatron_lm_config ipex_configuse_cpu rdzv_backend same_networkcommandsZ command_filetpu_envtpu_nametpu_vmtpu_zone tpu_use_sudotpu_use_cluster dynamo_configdebug).rrrZ MULTI_GPUZ MULTI_NPUZ MULTI_XPUZ MULTI_CPUrr%listrangerNOr rrrrrZ DEEPSPEEDrAssertionErrorstrfloatr Exceptionr ZFSDPr r r rZ MEGATRON_LMsplitreplace ValueErrorrospathabspathprintappendrrrZ LOCAL_MACHINE)0r4r8r9r5r6r:r;rBrCrLrAr@rKZ use_dynamoprefixZuse_custom_optionsZuse_mpsr=Z use_deepspeedZuse_deepspeed_configZuse_gradient_clippingZlauncher_queryZis_exclusion_filterZis_inclusion_filterr>Zuse_fsdpZsharding_strategy_queryZfsdp_wrap_queryZuse_no_split_modulesZfsdp_backward_prefetch_queryZfsdp_state_dict_type_queryr?Zuse_megatron_lmZ tpu_commandsZtpu_command_fileZtpu_downcast_bf16rErFrGrHrIrJZ machine_typer7r< run_commandsZuse_command_fileZanother_commandrr-r get_cluster_input0sn                                          r^)rWutilsrrrrrrr Zutils.constantsr r r r rrZ config_argsrZ config_utilsrrrrrrrr^rrrr s $  $