U 0-e'@sddlZddlZddlZddlZddlmZddlmZddlm Z GdddZ Gdd d Z Gd d d eZ Gd d d e Z GdddZGdddZdS)N)deepcopy)AcceleratedOptimizer)AcceleratedSchedulerc@sdeZdZdZddZddZddZdd d Zdd d ZddZ ddZ ddZ ddZ ddZ dS)HfDeepSpeedConfigaJ This object contains a DeepSpeed configuration dictionary and can be quickly queried for things like zero stage. A `weakref` of this object is stored in the module's globals to be able to access the config from areas where things like the Trainer object is not available (e.g. `from_pretrained` and `_get_resized_embeddings`). Therefore it's important that this object remains alive while the program is still running. [`Trainer`] uses the `HfTrainerDeepSpeedConfig` subclass instead. That subclass has logic to sync the configuration with values of [`TrainingArguments`] by replacing special placeholder values: `"auto"`. Without this special logic the DeepSpeed configuration is not modified in any way. Args: config_file_or_dict (`Union[str, Dict]`): path to DeepSpeed config file or dict. c Cst|trt|}n|tj|rHtj|ddd}t |}W5QRXnHzt | d}t |}Wn(tttfk rtd|YnX||_|dS)Nrzutf-8)encodingzoExpected a string path to an existing deepspeed config, or a dictionary, or a base64 encoded string. Received: ) isinstancedictrospathexistsioopenjsonloadbase64urlsafe_b64decodedecodeloadsUnicodeDecodeErrorAttributeError ValueErrorconfigset_stage_and_offload)selfZconfig_file_or_dictrfZconfig_decodedr[/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/accelerate/utils/deepspeed.py__init__*s    zHfDeepSpeedConfig.__init__cCsb|dd|_d|_|s$|r^tddg}t|d|dg}t||@dkr^d |_dS) Nzzero_optimization.stageFcpuZnvmez*zero_optimization.offload_optimizer.devicez&zero_optimization.offload_param.devicerT) get_value_stage_offloadis_zero2is_zero3setlen)rZoffload_devices_validZoffload_devicesrrrr?s z'HfDeepSpeedConfig.set_stage_and_offloadcCsH|j}|d}|}|D]"}||}|dkrd|fSq||fS)N.)rsplitpopget)r ds_key_longrnodesds_keynoderrrfind_config_nodeRs  z"HfDeepSpeedConfig.find_config_nodeNcCs&||\}}|dkr|S|||S)zG Returns the set value or `default` if no value is set N)r1r,)rr-defaultrr/rrrr"_szHfDeepSpeedConfig.get_valueFcCsh|j}|d}|D]<}|}||}|dkr|rJtd|d|jqdSq|dk rd||dS)z Deletes a sub-section of the config file if it's found. Unless `must_exist` is `True` the section doesn't have to exist. r)Nz Can't find z entry in the config: )rr*r,rr+)rr-Z must_existrr.r0Z parent_configrrrdel_config_sub_treehs  z%HfDeepSpeedConfig.del_config_sub_treecCs||}|dkrdSt|S)z Returns `True`/``False` only if the value is set, always `False` otherwise. So use this method to ask the very specific question of whether the value is set to `True` (and it's not set to `False`` or isn't set). NFr"boolrr-valuerrris_trues zHfDeepSpeedConfig.is_truecCs ||}|dkrdSt| S)z Returns `True`/``False` only if the value is set, always `False` otherwise. So use this method to ask the very specific question of whether the value is set to `False` (and it's not set to `True`` or isn't set). NFr4r6rrris_falses zHfDeepSpeedConfig.is_falsecCs |jdkS)Nrr#rrrrr%szHfDeepSpeedConfig.is_zero2cCs |jdkS)Nr:r;rrrr&szHfDeepSpeedConfig.is_zero3cCs|jSN)r$r;rrr is_offloadszHfDeepSpeedConfig.is_offload)N)F)__name__ __module__ __qualname____doc__rrr1r"r3r8r9r%r&r>rrrrrs  rc@s eZdZdZddZddZdS)DeepSpeedEngineWrapperz Internal wrapper for deepspeed.runtime.engine.DeepSpeedEngine. This is used to follow conventional training loop. Args: engine (deepspeed.runtime.engine.DeepSpeedEngine): deepspeed engine to wrap cCs ||_dSr=)engine)rrDrrrrszDeepSpeedEngineWrapper.__init__cKs|jj|f||jdSr=)rDbackwardstep)rZlosskwargsrrrrEs zDeepSpeedEngineWrapper.backwardN)r?r@rArBrrErrrrrCsrCcs>eZdZdZfddZd ddZddZed d ZZ S) DeepSpeedOptimizerWrapperz Internal wrapper around a deepspeed optimizer. Args: optimizer (`torch.optim.optimizer.Optimizer`): The optimizer to wrap. cs$tj|dddt|jd|_dS)NF)Zdevice_placementZscaleroverflow)superrhasattr optimizer__has_overflow__)rrL __class__rrrsz"DeepSpeedOptimizerWrapper.__init__NcCsdSr=r)rZ set_to_nonerrr zero_gradsz#DeepSpeedOptimizerWrapper.zero_gradcCsdSr=rr;rrrrFszDeepSpeedOptimizerWrapper.stepcCs|jr|jjSdS)zTWhether or not the optimizer step was done, or skipped because of gradient overflow.F)rMrLrIr;rrrstep_was_skippedsz*DeepSpeedOptimizerWrapper.step_was_skipped)N) r?r@rArBrrPrFpropertyrQ __classcell__rrrNrrHs   rHcs(eZdZdZfddZddZZS)DeepSpeedSchedulerWrapperz Internal wrapper around a deepspeed scheduler. Args: scheduler (`torch.optim.lr_scheduler.LambdaLR`): The scheduler to wrap. optimizers (one or a list of `torch.optim.Optimizer`): cst||dSr=)rJr)r schedulerZ optimizersrNrrrsz"DeepSpeedSchedulerWrapper.__init__cCsdSr=rr;rrrrFszDeepSpeedSchedulerWrapper.step)r?r@rArBrrFrSrrrNrrTs rTc@seZdZdZdddZdS) DummyOptima Dummy optimizer presents model parameters or param groups, this is primarily used to follow conventional training loop when optimizer config is specified in the deepspeed config file. Args: lr (float): Learning rate. params (iterable): iterable of parameters to optimize or dicts defining parameter groups weight_decay (float): Weight decay. **kwargs: Other arguments. MbP?rcKs||_||_||_||_dSr=)paramslr weight_decayrG)rrXrYrZrGrrrrszDummyOptim.__init__N)rWrr?r@rArBrrrrrrVsrVc@seZdZdZdddZdS)DummySchedulera Dummy scheduler presents model parameters or param groups, this is primarily used to follow conventional training loop when scheduler config is specified in the deepspeed config file. Args: optimizer (`torch.optim.optimizer.Optimizer`): The optimizer to wrap. total_num_steps (int, *optional*): Total number of steps. warmup_num_steps (int, *optional*): Number of steps for warmup. lr_scheduler_callable (callable, *optional*): A callable function that creates an LR Scheduler. It accepts only one argument `optimizer`. **kwargs: Other arguments. NrcKs"||_||_||_||_||_dSr=)rLtotal_num_stepswarmup_num_stepslr_scheduler_callablerG)rrLr]r^r_rGrrrr s zDummyScheduler.__init__)NrNr[rrrrr\sr\)rrrr copyrrLrrUrrrCrHrTrVr\rrrrs