U 0-ef@s*ddlZddlZddlmZddlmZddlZddlmZddl m Z ddl m Z dd l mZdd lmZmZmZdd lmZdd lmZed drddlmmZerddlmZddZd'edddZ ddZ!d(ddZ"eddZ#eddZ$ddZ%d d!Z&d)e'ed"d#d$Z(d%d&Z)dS)*N)contextmanager) MethodType)write_basic_config) PartialState)FSDP_PYTORCH_VERSION)DistributedType)is_deepspeed_availableis_safetensors_availableis_tpu_available) convert_model)is_torch_versionF)Z check_device) save_filecCs(tddsttdsdSt|tjjjS)zD Check whether the module was compiled with torch.compile() =)FullyShardedDataParallelforwardZ_original_forwardN __wrapped__Z _converted_to_transformer_engineF)Zto_transformer_engine)rnnparallelZDistributedDataParallelZ DataParallelrZ _orig_modr Z deepspeedrrrZ2torch.distributed.fsdp.fully_sharded_data_parallelrrrgetattr__dict__poprrrrr ) modelroptionsZ is_compiledZcompiled_modelrZFSDPrZoriginal_forwardrrrextract_model_from_parallel/s8            r%cCstdS)a  Introduces a blocking point in the script, making sure all processes have reached this point before continuing. Make sure all processes will reach this instruction otherwise one of your processes will hang forever. N)rwait_for_everyonerrrrr&cs r&cCsPtjtjkrt||n0tjdkrL|r@t||ddidn t||dS)a Save the data to disk. Use in place of `torch.save()`. Args: obj: The data to save f: The file (or file-like object) to use to save the data safe_serialization (`bool`, *optional*, defaults to `False`): Whether to save `obj` using `safetensors` rformatpt)metadataN) rZdistributed_typer ZTPUxmsaveZlocal_process_indexsafe_save_filer)objfZsafe_serializationrrrr+ps  r+ccstj}tt_dV|t_dS)a A context manager that will cache origin `os.environ` and replace it with a empty dictionary in this context. When this context exits, the cached `os.environ` will be back. Example: ```python >>> import os >>> from accelerate.utils import clear_environment >>> os.environ["FOO"] = "bar" >>> with clear_environment(): ... print(os.environ) ... os.environ["FOO"] = "new_bar" ... print(os.environ["FOO"]) {} new_bar >>> print(os.environ["FOO"]) bar ``` N)osenvirondict)Z_old_os_environrrrclear_environmentsr2cksi}|D]6\}}|}|tjkr4tj|||<t|tj|<q dV|D]2}|}||krr||tj|<qNtj|dqNdS)a A context manager that will add each keyword argument passed to `os.environ` and remove them when exiting. Will convert the values in `kwargs` to strings and upper-case all the keys. Example: ```python >>> import os >>> from accelerate.utils import patch_environment >>> with patch_environment(FOO="bar"): ... print(os.environ["FOO"]) # prints "bar" >>> print(os.environ["FOO"]) # raises KeyError ``` N)itemsupperr/r0strr")kwargsZ existing_varskeyvaluerrrpatch_environments r9cCsHt|ds t|ds t|d|}t|dr0|jSt|dr@|jSt|S)z( Gets a pretty name from `obj`. __qualname____name__ __class__)rr r:r;r5)r-rrrget_pretty_names   r=cCs@|D]2\}}t|tr2||i}t||q|||<q|S)z Recursively merges two dictionaries. Args: source (`dict`): The dictionary to merge into `destination`. destination (`dict`): The dictionary to merge `source` into. )r3rr1 setdefault merge_dicts)source destinationr7r8noderrrr?s     r?)portreturnc CsF|dkr d}ttjtj }|d|fdkW5QRSQRXdS)z Checks if a port is in use on `localhost`. Useful for checking if multiple `accelerate launch` commands have been run and need to see if the port is already in use. Nis6           4   #