U 9%e?E@sdZddlmZddlZddlZddlmZddl m Z ddl m Z m Z mZe rfddlZddlmZeeZdd Ze rerdd lmZn dd lmZGd d d eZGdddeZdaddZddZddZddZ ddZ!dddZ"ddZ#dS) z Integration with Deepspeed N) partialmethod)dep_version_check)is_accelerate_availableis_torch_availablelogging) get_schedulercCsBtjddk }|r>ztd}WdStjk r<YdSXdS)N deepspeedTF) importlibutil find_specimportlib_metadatametadataZPackageNotFoundError)Zpackage_exists_rb/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/transformers/integrations/deepspeed.pyis_deepspeed_available#s r)HfDeepSpeedConfig)objectcs eZdZdZfddZZS)raJ This object contains a DeepSpeed configuration dictionary and can be quickly queried for things like zero stage. A `weakref` of this object is stored in the module's globals to be able to access the config from areas where things like the Trainer object is not available (e.g. `from_pretrained` and `_get_resized_embeddings`). Therefore it's important that this object remains alive while the program is still running. [`Trainer`] uses the `HfTrainerDeepSpeedConfig` subclass instead. That subclass has logic to sync the configuration with values of [`TrainingArguments`] by replacing special placeholder values: `"auto"`. Without this special logic the DeepSpeed configuration is not modified in any way. Args: config_file_or_dict (`Union[str, Dict]`): path to DeepSpeed config file or dict. cs(t|tdtdt|dS)NZ accelerater )set_hf_deepspeed_configrsuper__init__selfZconfig_file_or_dict __class__rrrIszHfDeepSpeedConfig.__init__)__name__ __module__ __qualname____doc__r __classcell__rrrrr8srcsVeZdZdZfddZddZddZdd d Zeed d Z ddZ ddZ Z S)HfTrainerDeepSpeedConfigz The `HfTrainerDeepSpeedConfig` object is meant to be created during `TrainingArguments` object creation and has the same lifespan as the latter. cst|d|_g|_dSN)rr_dtype mismatchesrrrrrWs z!HfTrainerDeepSpeedConfig.__init__cCs|jdkrtd|jS)Nz8trainer_config_process() wasn't called yet to tell dtype)r# ValueErrorrrrrdtype\s zHfTrainerDeepSpeedConfig.dtypecCs"||}|dkrdS|dkSdS)NFauto) get_value)r ds_key_longvalrrris_autoas z HfTrainerDeepSpeedConfig.is_autoNTc Cs~||\}}|dkrdS||dkr4|||<dS|srrrtrainer_config_processsD      z/HfTrainerDeepSpeedConfig.trainer_config_processcsdddg}fdd|D}t|dkrt|jdr>|jj}n*t|jdrXt|jj}ntd |d d||rdd ||dd | d |d d| |dtj dkrd j }td|ddS)z This stage is run after we have the model and know num_training_steps. Now we can complete the configuration process. z$zero_optimization.reduce_bucket_sizez-zero_optimization.stage3_prefetch_bucket_sizez4zero_optimization.stage3_param_persistence_thresholdcsg|]}|r|qSr)r,).0xr&rr s zDHfTrainerDeepSpeedConfig.trainer_config_finalize..r hidden_size hidden_sizeszThe model's config file has neither `hidden_size` nor `hidden_sizes` entry, therefore it's not possible to automatically fill out the following `auto` entries in the DeepSpeed config file: zb. You can fix that by replacing `auto` values for these keys with an integer value of your choice.g? z scheduler.params.total_num_stepsznum_training_steps (calculated)z!scheduler.params.warmup_num_stepsZ warmup_steps z]Please correct the following DeepSpeed config values that mismatch TrainingArguments values: zF The easiest method is to set these DeepSpeed config values to 'auto'.N) lenhasattrr1rEmaxrFr%r=is_zero3r2get_warmup_stepsr$join)rr@modelnum_training_stepsZhidden_size_based_keysZhidden_size_auto_keysrEr$rr&rtrainer_config_finalizes0        z0HfTrainerDeepSpeedConfig.trainer_config_finalize)NT) rrrrrr'r,r2rr=rArQr rrrrr!Qs   :r!cCst|adSr")weakrefref_hf_deepspeed_config_weak_ref)Zhf_deepspeed_config_objrrrrsrcCsdadSr")rTrrrrunset_hf_deepspeed_configsrUcCs$tdk rtdk rtSdSdS)NF)rTrLrrrris_deepspeed_zero3_enableds rVcCs"tdk rtdk rtjSdSdSr")rTr1rrrrdeepspeed_configsrWc sddlm}m}|j}d}d|kr<|jr0td||d}n"|rNtd }d|d <d} d |krt||} n4t ||rfd d } ||| d } nj |d} || fS)zY A convenience wrapper that deals with optimizer and lr scheduler configuration. r) DummyOptimDummySchedulerN optimizerz|--adafactor was passed, but also found `optimizer` configured in the DeepSpeed config. Only one optimizer can be configured.)paramszDetected ZeRO Offload and non-DeepSpeed optimizers: This combination should work as long as the custom optimizer has both CPU and GPU implementation (except LAMB)TZzero_allow_untested_optimizerZ schedulercstjj|jdS)N)rZZnum_warmup_stepsrP)rr@Zlr_scheduler_typerM)rZrPtrainerrr_lr_scheduler_callable@s  z5deepspeed_optim_sched.._lr_scheduler_callable)Zlr_scheduler_callable)rPrZ) Zaccelerate.utilsrXrYr1Z adafactorr%Z is_offloadloggerinfoZcreate_optimizer isinstanceZcreate_scheduler) r]hf_deepspeed_configr@rPmodel_parametersrXrYr1rZ lr_schedulerr^rr\rdeepspeed_optim_scheds.   reFc Csddlm}|j}|j}|jjjj}||||| | |rv| sTt d| d| dd\}}d} n0d|_ttdd |} t||||| \}}||fS) aj Init DeepSpeed, after updating the DeepSpeed configuration with any relevant Trainer's args. If `resume_from_checkpoint` was passed then an attempt to resume from a previously saved checkpoint will be made. Args: trainer: Trainer object num_training_steps: per single gpu resume_from_checkpoint: path to a checkpoint if to resume from after normal DeepSpeedEngine load inference: launch in inference mode (no optimizer and no lr scheduler) Returns: optimizer, lr_scheduler We may use `deepspeed_init` more than once during the life of Trainer, when we do - it's a temp hack based on: https://github.com/microsoft/DeepSpeed/issues/1394#issuecomment-937405374 until Deepspeed fixes a bug where it can't resume from a checkpoint after it did some stepping https://github.com/microsoft/DeepSpeed/issues/1612 r)r_zMZeRO inference only makes sense with ZeRO Stage 3 - please adjust your configrZrd)NNNcSs|jSr")Z requires_grad)prrr{z deepspeed_init..)Zdeepspeed.utilsr_rOr@Z acceleratorstateZdeepspeed_pluginZ hf_ds_configrQsetLevelZget_process_log_levelrLr%Zdel_config_sub_treerZlistfilter parametersre) r]rPZ inferenceZ ds_loggerrOr@rbrZrdrcrrrdeepspeed_initOs.    rncCsvddl}t||d}t|dkrdtd||j|ddd\}}|dkrrtd|ntd|dS)Nrz /global_step*zAttempting to resume from T)Zload_optimizer_statesZload_lr_scheduler_statesz-[deepspeed] failed to resume from checkpoint z!Can't find a valid checkpoint at )globsortedrIr_r`Zload_checkpointr%)Zdeepspeed_engineZcheckpoint_pathroZdeepspeed_checkpoint_dirsZ load_pathrrrrdeepspeed_load_checkpoints  rq)F)$rimportlib.metadatarr importlib.utilr rR functoolsrZdependency_versions_checkrutilsrrrr? optimizationrZ get_loggerrr_rZaccelerate.utils.deepspeedrZDeepSpeedConfigbuiltinsrr!rTrrUrVrWrernrqrrrrs2       A 7