U 0-et!@sddlZddlZddlmZddlmZddlZddlZddl m Z ddl m Z m Z mZmZmZmZmZmZmZeddrddlmmZdd lmZdd lmZeeZdeee e!e!e"e d d d Z#dddZ$de"dddZ%de"dddZ&dS)N)Path)List) GradScaler) MODEL_NAMEOPTIMIZER_NAMERNG_STATE_NAME SCALER_NAMESCHEDULER_NAMEget_pretty_nameis_tpu_availableis_xpu_availablesaveF)Z check_device) get_logger) PartialState) output_dir model_states optimizers schedulers process_indexscalercCst|D]R\}}|dkr"tdntd|d}tj||} t|| td| qt|D]Z\}} | }|dkrt dnt d|d} tj|| } t|| td| qdt|D]Z\}} | }|dkrt dnt d|d}tj||}t||td|q|dk r`|}tj|t }t ||td|i}t d|d }t|d <tj|d <t |d <trt j|d <nt j|d<trt|d<tj||}t ||td||S)a Saves the current states of the models, optimizers, scaler, and RNG generators to a given directory. Args: output_dir (`str` or `os.PathLike`): The name of the folder to save all relevant weights and states. model_states (`List[torch.nn.Module]`): A list of model states optimizers (`List[torch.optim.Optimizer]`): A list of optimizer instances schedulers (`List[torch.optim.lr_scheduler._LRScheduler]`): A list of learning rate schedulers process_index (`int`): The current process index in the Accelerator state scaler (`torch.cuda.amp.GradScaler`, *optional*): An optional gradient scaler instance to save r.bin_zModel weights saved in zOptimizer state saved in zScheduler state saved in NzGradient scaler state saved in .pkl random_statenumpy_random_seedtorch_manual_seedtorch_xpu_manual_seedtorch_cuda_manual_seedxm_seedzRandom states saved in ) enumeraterospathjoinrloggerinfo state_dictrr r torchrrandomgetstatenpZ get_stateZ get_rng_stater xpuZget_rng_state_allcudar xm)rrrrrristate weights_nameZoutput_model_fileoptoptimizer_nameZoutput_optimizer_file schedulerscheduler_nameZoutput_scheduler_fileZoutput_scaler_filestatesZ states_nameZoutput_states_filer6Y/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/accelerate/checkpointing.pysave_accelerator_state/sH" " "       r8c KsP|dkrtd|dkrd}n|dkr.tj}t|D]T\}} |dkrPtdntd|d} tj|| } ||jt j | |d f|q6t d t|D]T\}} |dkrt dnt d|d} tj|| }t j ||d }|||qt d t|D]L\}}|dkr"tdntd|d}tj||}|t |qt d |dk rtj|t}|t |t d zt tj|td|d}t|dtj|dt |dtrt j|dnt j|dtrt|dt dWn tk rJt dYnXdS)a Loads states of the models, optimizers, scaler, and RNG generators from a given directory. Args: input_dir (`str` or `os.PathLike`): The name of the folder to load all relevant weights and states. models (`List[torch.nn.Module]`): A list of model instances optimizers (`List[torch.optim.Optimizer]`): A list of optimizer instances schedulers (`List[torch.optim.lr_scheduler._LRScheduler]`): A list of learning rate schedulers process_index (`int`): The current process index in the Accelerator state scaler (`torch.cuda.amp.GradScaler`, *optional*): An optional *GradScaler* instance to load map_location (`str`, *optional*): What device to load the optimizer state onto. Should be one of either "cpu" or "on_device". load_model_func_kwargs (`dict`, *optional*): Additional arguments that can be passed to the model's `load_state_dict` method. )Ncpu on_devicezaUnsupported optimizer map location passed, please choose one of `None`, `'cpu'`, or `'on_device'`Nr9r:rrr map_locationz%All model weights loaded successfullyz(All optimizer states loaded successfullyz(All scheduler states loaded successfullyz$GradScaler state loaded successfullyrrrrrrrz%All random states loaded successfullyzCould not load random states) TypeErrorrZdevicer rr!r"r#load_state_dictr'loadr$r%rr r rr(setstater*Z set_stateZ set_rng_stater r+Zset_rng_state_allr,r r- Exception)Z input_dirmodelsrrrrr<Zload_model_func_kwargsr.modelr0Zinput_model_filer1r2Zinput_optimizer_fileZoptimizer_stater3r4Zinput_scheduler_fileZinput_scaler_filer5r6r6r7load_accelerator_statetsR" " $    rD)indexcCsBt|d|d}tdt|d|t||dS)zL Saves the state of `obj` to `{path}/custom_checkpoint_{index}.pkl` Zcustom_checkpoint_rzSaving the state of z to N)rr$r%r r'rr&)objr"rEZ save_locationr6r6r7save_custom_statesrGcCsB|d|d}tdt|d||tj|dddS)zL Loads the state of `obj` at `{path}/custom_checkpoint_{index}.pkl` z/custom_checkpoint_rzLoading the state of z from r9r;N)r$r%r r>r'r?)rFr"rEZ load_locationr6r6r7load_custom_statesrH)N)NN)r)r)'r!r(pathlibrtypingrnumpyr*r'Ztorch.cuda.amprutilsrrrr r r r r rZtorch_xla.core.xla_modelcoreZ xla_modelr-loggingrr/r__name__r$strdictlistintr8rDrGrHr6r6r6r7s4   ,     K T