U 0-e2 @sddlZddlZddlZddlZddlZddlZddlZddlZddlZddl m Z ddl m Z m Z mZmZmZddlZddlmZddlmZddlmZmZddlmZmZmZdd lmZmZm Z m!Z!dd l"m#Z#m$Z$m%Z%dd l&m'Z'm&Z&ed d rddl(Z(e r$ddl)m*Z*ddl+m,Z-dZ.e/e0Z1ee2e3fdddZ4ej5dddZ6ej7eej8e2e2fdddZ9defe e3ej7fee2e3fe3dddZ:dWej;e3ee2e3ej8feej7eee3ej5feej<ddd Z=dXej;e>e>d"d#d$Z?Gd%d&d&e@ZAej;d'd(d)ZBd*d+ZCd,d-ZDej;d'd.d/ZEd0d1ZFee3ej8fej5d2d3d4ZGdYej;eee3ej8fee e3ee3ej8ffd5d6d7ZHe ee3ejj;fe e3e2fe e3d8d9d:ZIdZee ee2e3fee2e3ffd;ddDdEdFZMejj;d'dGdHZNd]ej;ee ee2e3fee2e3ffee e3eee3ej5fee e3ee3ej5ffe>dIdJdKZOej;e e3ee2e3ej8ffdLdMdNZPd^dOdPZQd_ej;ee3ejRfee e3ee2e3ej8ffeee3ejRfeee3ej5fe>e>e e3e>dQ dRdSZSd`e>edTdUdVZTdS)aN) defaultdict)DictListOptionalTupleUnion)AcceleratorState)SAFE_WEIGHTS_NAME WEIGHTS_NAME)AutocastKwargs CustomDtypeDistributedType)is_mps_availableis_npu_availableis_safetensors_availableis_xpu_available)load_offloaded_weightoffload_weightsave_offload_index)is_tqdm_availabletqdmF)Z check_device) safe_open) load_filezpytorch_model.bin.index.json)sizecCsd}d|d}zPt|tr&|}n:|drPtt|ddd}n|drxtt|ddd }n|d rtt|ddd }n|d rtt|dd d}|dr|dn|}n|dr tt|dd d}|dr|dn|}n@|dr`tt|dd d}|dr\|dn|}Wntk rt|YnX|dkrt||S)a( Converts a size expressed as a string with digits an unit (like `"5MB"`) to an integer (in bytes). Args: size (`int` or `str`): The size to convert. Will be directly returned if an `int`. Example: ```py >>> convert_file_size_to_int("1MiB") 1048576 ``` rz`size` z] is not in a valid format. Use an integer for bytes, or a string with an unit (like '5.0GB').ZGIBNi@ZMIBiZKIBiGBiʚ;bMBi@BZKBi) isinstanceintupperendswithfloat ValueError)rZmem_sizeerr_msgZint_sizer)Z/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/accelerate/utils/modeling.pyconvert_file_size_to_int4s4   r+dtypecCsj|tjkrdS|tjkrdS|tjkr*dStdt|}|dkrRtd|dt | d}|d S) z Returns the size (in bytes) occupied by one parameter of type `dtype`. Example: ```py >>> dtype_byte_size(torch.float32) 4 ``` g?g?r z [^\d](\d+)$Nz`dtype` is not a valid dtype: .rr ) torchboolrZINT4ZFP8researchstrr'r#groups)r-Z bit_searchZbit_sizer)r)r*dtype_byte_size`s   r5)tensorreturncCstjdtjdtjdtjdtjdtjdtjdtjdtj dtj di }z| }| }Wnftk rz&| }|||j}Wn*tk rd}|||j}YnXYnX|j||fS)a Unique identifier to a tensor storage. Multiple different tensors can share the same underlying storage. For example, "meta" tensors all share the same storage, and thus their identifier will all be equal. This identifier is guaranteed to be unique and constant for this tensor's storage during its lifetime. Two tensor storages with non-overlapping lifetimes may have the same id. r rr r)r/Zint64float32Zint32bfloat16float16Zint16Zuint8int8r0Zfloat64Zuntyped_storageZdata_ptrnbytes ExceptionZstoragerr-NotImplementedErrorZnelementdevice)r6Z_SIZEZ storage_ptrZ storage_sizer)r)r*id_tensor_storagexs@   rAZ10GB) state_dictmax_shard_size weights_namec Cstt|}ig}d}d}i}|D]\}}t|tr8q"nt|} | |kr^|| } ||| |<q"|t|j} || |kr|id}||d|<|| 7}|| 7}t |d|| <q"t |dkr||didfSi} i} t |D]t\}}| dd|dddt |dd}| d d|dddt |dd }|| |<| D]}|| |<qHqd |i}|| d }| |fS) a Splits a model state dictionary in sub-checkpoints so that the final size of each sub-checkpoint does not exceed a given size. The sub-checkpoints are determined by iterating through the `state_dict` in the order of its keys, so there is no optimization made to make each sub-checkpoint as close as possible to the maximum size passed. For example, if the limit is 10GB and we have weights of sizes [6GB, 6GB, 2GB, 6GB, 2GB, 2GB] they will get sharded as [6GB], [6+2GB], [6+2+2GB] and not [6+2+2GB], [6+2GB], [6GB]. If one of the model's weight is bigger that `max_sahrd_size`, it will end up in its own sub-checkpoint which will have a size greater than `max_shard_size`. Args: state_dict (`Dict[str, torch.Tensor]`): The state dictionary of a model to save. max_shard_size (`int` or `str`, *optional*, defaults to `"10GB"`): The maximum size of each sub-checkpoint. If expressed as a string, needs to be digits followed by a unit (like `"5MB"`). weights_name (`str`, *optional*, defaults to `"pytorch_model.bin"`): The name of the model save file. rr Nz.bin-Z05dz-of- .safetensors total_size)metadata weight_map) r+itemsr"r3rAnumelr5r-appendlen enumeratereplacekeys)rBrCrDZsharded_state_dictsZlast_block_sizerHZstorage_id_to_blockkeyweightZ storage_idZblock_idZ weight_sizerJZshardsidxZshardZ shard_filerIindexr)r)r*shard_checkpointsJ      &  rV)module tensor_namer@valuer-fp16_statisticsc Csd|krV|d}|ddD].}t||}|dkrHt|d|d|}q|d}||jkr~||jkr~t|d|d||jk} t||} | jtdkr|dtdfkr|dkrt|d|d|dk r<| j|jkrtd|jd |d | jd |dkr || j }nt |j  d s<||}||jkrR|j|nd} t | } d} t &| dk r| jj d krt|j d kr| jdkr|} d}|dkr| |}|dk r,|dtdfkr,||}| s,| || jd|j|<n(t|tjr||}ntj||d}| dk r:| }| rN||j|<n@|dk spt||j|jkrt |j|} |j|j}| jdkr0| jdkr|j tjkr|tj}|dkr| jdkr| |fd| ji|dd}|jd|_|jd|_n| |fd| ji||}n| || jd|}||j|<|dk rrt|j|d||~|jjdkr"t|jdddkr"t |jjdkr"t|j d krt|jnd}t|jdds|dk r|jdk r |jjj dkr ||}n|jdkr||}nl|jjdkrt|jdddkrt|j d krbt|jnd}t|jdds|dk r|j||_W5QRXtjdS)a A helper function to set a given tensor (parameter of buffer) of a module on a specific device (note that doing `param.to(device)` creates a new tensor not linked to the parameter, which is why we need this function). Args: module (`torch.nn.Module`): The module in which the tensor we want to move lives. param_name (`str`): The full name of the parameter/buffer. device (`int`, `str` or `torch.device`): The device on which to set the tensor. value (`torch.Tensor`, *optional*): The value of the tensor (useful when going from the meta device to any other device). dtype (`torch.dtype`, *optional*): If passed along the value of the parameter will be cast to this `dtype`. Otherwise, `value` will be cast to the dtype of the existing parameter in the model. fp16_statistics (`torch.HalfTensor`, *optional*): The list of fp16 statistics to set on the module, used for 8 bit model serialization. r.NrEz has no attribute z- does not have a parameter or a buffer named metaz7 is on the meta device, we need a `value` to put in on z Trying to set a tensor of shape z in "z" (which has shape z), this look incorrect.)z torch.uintz torch.intz torch.boolcuda) Int8ParamsZ FP4Paramscpu) requires_gradr@r]r_rSCBZ Linear8bitLtZ Linear4bitZ quant_state)splitgetattrr' _parameters_buffersr@r/shapetor-r3 startswithtypeZno_grad__name__r_r"Tensorr6__dict__r9r;ZCBrasetattr __class__rSrUZbiasr\Z empty_cache)rWrXr@rYr-rZsplitsrbZ new_moduleZ is_buffer old_valueparamZ param_clsZdevice_quantization new_valuekwargsZ device_indexr)r)r*set_module_tensor_to_devices    *          "   "   "   ""rtT)rWinclude_buffersrecurseccs8|j|dD] }|Vq |r4|j|dD] }|Vq(dS)aq A helper function that gathers all the tensors (parameters + buffers) of a given module. If `include_buffers=True` it's the same as doing `module.named_parameters(recurse=recurse) + module.named_buffers(recurse=recurse)`. Args: module (`torch.nn.Module`): The module we want the tensors on. include_buffer (`bool`, *optional*, defaults to `True`): Whether or not to include the buffers in the result. recurse (`bool`, *optional`, defaults to `False`): Whether or not to go look in every submodule or just return the direct parameters and buffers. rvN)named_parameters named_buffers)rWrurvZnamed_parameterZ named_bufferr)r)r*named_module_tensorsps rzcs(eZdZdZfddZddZZS)FindTiedParametersResultz This is a subclass of a list to handle backward compatibility for Transformers. Do not rely on the fact this is not a list or on the `values` method as in the future this will be removed. cstj||dSN)super__init__)selfargsrsrnr)r*r~sz!FindTiedParametersResult.__init__cCstdd|DgS)NcSsg|]}|ddqS)r Nr)).0xr)r)r* sz3FindTiedParametersResult.values..)sum)rr)r)r*valuesszFindTiedParametersResult.values)rj __module__ __qualname____doc__r~r __classcell__r)r)rr*r{s r{)modelcCsd}d}d}dddt|jDkrt|doDt|jddoD|}t|dojt|jddojt|jdd}td d |D}t|||gS) z Check if there is any indication in the given model that some weights should be tied. Args: model (`torch.nn.Module`): The model to inspect Returns: bool: True if the model needs to have tied weights FZPreTrainedModelcSsg|] }|jqSr))rj)rcr)r)r*rsz3check_tied_parameters_in_config..configZtie_word_embeddingsZis_encoder_decoderZtie_encoder_decodercss|]}t|dVqdS)Z _tie_weightsN)hasattr)rrWr)r)r* sz2check_tied_parameters_in_config..) inspectgetmrornrrcrZget_output_embeddingsanymodules)rZhas_tied_word_embeddingZhas_tied_encoder_decoderZhas_tied_moduler)r)r*check_tied_parameters_in_configs      rcCsP||kr||Sd|ddd}||krBtd|dn t||SdS)Nr.rEz-The `device_map` does not contain the module )joinrbr'_get_param_device)rq device_mapZ parent_paramr)r)r*rs rcCsP|D]F}i}|D]}t||||<qtt|dkrtd|dqdS)a9 Check if tied parameters are on the same device Args: tied_params (`List[List[str]]`): A list of lists of parameter names being all tied together. device_map (`Dict[str, Union[int, str, torch.device]]`): A map that specifies where each submodule should go. r z*Tied parameters are on different devices: zC. Please modify your custom device map or set `device_map='auto'`. N)rrNsetrloggerwarn) tied_paramsrZ tie_paramZtie_param_devicesrqr)r)r*$check_tied_parameters_on_same_devices  rc Ks|dd}|dd}|di}|dkr@dd|D}nl|D]b\}}|dkr\|n |d|}||krH|D].\}} | |krz||krg||<|||qzqH|D]2\}} |dkr|n |d|} t| || |d qtd d |DS) a Find the tied parameters in a given model. The signature accepts keyword arguments, but they are for the recursive part of this function and you should ignore them. Args: model (`torch.nn.Module`): The model to inspect. Returns: List[List[str]]: A list of lists of parameter names being all tied together. Example: ```py >>> from collections import OrderedDict >>> import torch.nn as nn >>> model = nn.Sequential(OrderedDict([("linear1", nn.Linear(4, 4)), ("linear2", nn.Linear(4, 4))])) >>> model.linear2.weight = model.linear1.weight >>> find_tied_parameters(model) [['linear1.weight', 'linear2.weight']] ``` rxNprefixresultcSsi|]\}}||qSr)r))rnpr)r)r* sz(find_tied_parameters..r.)rxrrcSs&g|]\}}t|gtt|qSr))sortedlistr)rrSZtiedr)r)r*r sz(find_tied_parameters..)getrxrKrMnamed_childrenfind_tied_parametersr{) rrsrxrrnameZ parameterZ full_namenew_name new_paramchildZ child_namer)r)r*rs"   rcCsn|D]d}d}|D]V}|}|d}|ddD]}t||}q.|dkrVt||d}qt||d|qqdS)aX Reties tied parameters in a given model if the link was broken (for instance when adding hooks). Args: model (`torch.nn.Module`): The model in which to retie parameters. tied_params (`List[List[str]]`): A mapping parameter name to tied parameter name as obtained by `find_tied_parameters`. Nr.rE)rbrcrm)rr tied_groupZ param_to_tie param_namerWrorbr)r)r*retie_parameterss   r)r-r7cCs$t|tr |dd}tt|}|S)z4 Just does torch.dtype(dtype) if necessary. torch.r)r"r3rPrcr/r,r)r)r*_get_proper_dtype&s   r)rr-special_dtypesc Cs|dk rt|}t|}|dk rDdd|D}dd|D}tt}t|ddD]\}}|dk r||kr|||}n4|dkr|t|j}n|t|t|j}| d} t t | dD]"} |d | d| |7<qqX|S) z> Compute the size of each submodule of a given model. NcSsi|]\}}|t|qSr))rrrRZdtypr)r)r*r=sz(compute_module_sizes..cSsi|]\}}|t|qSr))r5rr)r)r*r>sTrwr.r ) rr5rKrr#rzrLr-minrbrangerNr) rr-rZ dtype_sizeZspecial_dtypes_size module_sizesrr6rZ name_partsrTr)r)r*compute_module_sizes1s" "r)rrno_split_module_classesc sd}g}|}t|dkr|d\}t|tjjrDt|ng}t|dks`|j j |kr|}||kr||}g}q||kr| qfdd|D|}q||fS)aO Utility function that will scan a list of named modules and return the maximum size used by one full layer. The definition of a layer being: - a module with no direct children (just parameters and buffers) - a module whose class name is in the list `no_split_module_classes` Args: modules (`List[Tuple[str, torch.nn.Module]]`): The list of named modules where we want to determine the maximum layer size. module_sizes (`Dict[str, int]`): A dictionary mapping each layer name to its size (as generated by `compute_module_sizes`). no_split_module_classes (`List[str]`): A list of class names for layers we don't want to be split. Returns: `Tuple[int, List[str]]`: The maximum size of a layer with the list of layer names realizing that maximum size. rcs"g|]\}}d||fqSr.r)rrv module_namer)r*rqsz&get_max_layer_size..) copyrNpopr"r/nnModulerrrnrjrM) rrrmax_sizeZ layer_namesmodules_to_treatrWmodules_childrenrr)rr*get_max_layer_sizeNs  r max_memoryc sddl}dkrtjs&ts&intsjttjD]}tjdg|d}q:ddttjDnBttjD]}tjdt d|d}qxddttjDt r| j d<n| j d <SD]"}t |trt||<qd d D}|tr(tjntj}|D]8}||ksN|dkr6td |d tt|q6|fdd dD}D] } | |krtd | dqfdd|DS)zb Get the maximum memory available if nothing is passed, converts string to int otherwise. rNr`cSsi|]}|tj|dqSr)r/r\Z mem_get_inforir)r)r*rsz"get_max_memory..xpucSsi|]}|tj|qSr))r/rZmax_memory_allocatedrr)r)r*rsmpsr^cSsg|]}t|tr|qSr)r"r#rkr)r)r*rs z"get_max_memory..zDevice z) is not available, available devices are csg|]}|kr|qSr))rQrrr)r*rs )rr^diskzX is not recognized, available devices are integers(for GPU/XPU), 'mps', 'cpu' and 'disk'csi|]}||qSr)r)rrr)r*rs)psutilr/r\Z is_availablerrZ device_countr6rr@rZvirtual_memory availabler"r3r+rQsortrwarningrr') rrr_rRZ gpu_devices num_devicesr@Z all_devicesrr)rr*get_max_memoryus@"   rr)rrcsdkr dndfdd|D}tt|dkrrt|dkrrfdd|DD] }||=qZ|d|<fdd|D}tdkrtddndtfd d |D}|D]}t||d q|S) z] Cleans a device_map by grouping all submodules that go on the same device together. rr.csg|]\}}|r|qSr)rh)rrrrr)r*rs z$clean_device_map..r csg|]}|r|qSr)rrrr)r*rs rcs*g|]"}|rt|tkr|qSr))rhrNr)rrr)r*rs c3s&|]}d|ddVqdS)r.N)rrbr)rTr)r*rsz#clean_device_map..r)rKrNrrQrbclean_device_map)rrrrZchildren_modulesrr))rTrrr*rs "rc Cs|dkst|dkrdS|D]\}}d|kr2q d}d|kr||dd|kr||dd}ttj||d||}tj||d}t||}t||d||dq dS)a Loads the weights from the offload folder into the model. Args: model (`torch.nn.Module`): The model to load the weights into. index (`dict`): A dictionary containing the parameter name and its metadata for each parameter that was offloaded from the model. offload_folder (`str`): The folder where the offloaded weights are stored. NrrarSz.datr^)rYrZ) rNrKrPrQrospathrrt) rrUoffload_folderrrIrZZ weight_nameZ tensor_filerSr)r)r*load_offloaded_weightss   r)rrrr-rlow_zerocsdk}tts.tfddD}ntfddD}|dkrPS|dkrd}|rD]4}t|trh|d9<td |d qqht|||d d |r|dn|} |dkrg}nt|t t fs|g}t|dkri}  D]x\} } | d krq|} | d D]}t | |} q$| jj}||krZ|| krZ| | |<t| t|krqzqt| dkrt| nd}nd}fddDfdd DfddDttfddDttd}tdt||}| |7} t tdd D}|ddD]0}t|r^|dkr^dn| ||<q@|rtdd tfddtd|D}t|dd<S)aR Compute a `max_memory` dictionary for [`infer_auto_device_map`] that will balance the use of each available GPU. All computation is done analyzing sizes and dtypes of the model parameters. As a result, the model can be on the meta device (as it would if initialized within the `init_empty_weights` context manager). Args: model (`torch.nn.Module`): The model to analyze. max_memory (`Dict`, *optional*): A dictionary device identifier to maximum memory. Will default to the maximum memory available if unset. no_split_module_classes (`List[str]`, *optional*): A list of layer class names that should never be split across device (for instance any layer that has a residual connection). dtype (`str` or `torch.dtype`, *optional*): If provided, the weights will be converted to that type when loaded. special_dtypes (`Dict[str, Union[str, torch.device]]`, *optional*): If provided, special dtypes to consider for some specific weights (will override dtype used as default for all weights). low_zero (`bool`, *optional*): Minimizes the number of weights on GPU 0, which is convenient when it's used for other operations (like the Transformers generate function). Ncs,g|]$}t|jdkr|dkr|qS)r\r)r/r@rirdrr)r*rs z'get_balanced_memory..csFg|]>}|dkrt|jdks2tj|jdkr|dkr|qS)r^rZgpur)r/r@rirZget_device_propertiesZdev_typerrr)r*r s  rr Fg?z(We will use 90% of the memory on device z for storing the model, and 10% for the buffer to avoid OOM. You can set `max_memory` in to a higher value to use more memory (at your own risk).r-rrr.cs*g|]"tfddDdkrqS)cs&g|]}dks|dr|qSrr.rrrrr)r*rFs2get_balanced_memory...rrNrrrr*rFscsi|]\}}|kr||qSr)r)r)leavesr)r*rGsz'get_balanced_memory..cs*g|]"tfddDdkrqS)cs&g|]}dks|dr|qSrrrrr)r*rIsrrrrrrr*rIscsg|] }|qSr)r)rrrr)r*rJsg?css(|] \}}t|tr|dkr|VqdS)rNr)rZ device_idZ device_memr)r)r*rPs z&get_balanced_memory..rEcsg|] }|qSr)r)rrr)r*rYs)rrrNrQr"r#rinforrtuplerKrbrcrnrjrmaxrrrrr)rrrr-rrZuser_not_set_max_memoryrrRZper_gpuZno_split_childrenrr submoduleZsubmodule_name class_namebufferZ mean_leavesZ gpus_idx_listrTZmin_zeror))rrrr*get_balanced_memorysv$        (.*rcCsht|}t|dd}|dkr g}t|jddt|t|jdd}t|||}|d}||fS)z:Computes the total size of the model and its largest layerZ_no_split_modulesNFrwr)rrcrrxrryr)rsizesZno_split_modulesrZ largest_layerrHr)r)r*calculate_maximum_sizes_s   r)rrrr-rverbosec sLt|}|dkrg}nt|ttfs*|g}t|}d|krH|ddd|D}d|krfdg}n t|dkr|ddg}ndg}t|||d} t|} t |rt| dkrt d i} d} d} t|j d d t| t|jd d }t|| |\}}t|dkrD|d\}|r4td d fdd|D}t|dkrntdd|D| |\}}| }fdd| D}|rt|dkrtd|tfdd|Dg}|rt|dkrtd||| }|dkr||nd}|| |kr||}|dk r"| ||kr"t|tjrBgn t| }|r~td|| dd|| d|d t|dks|jj|kr|rtd| d7} |fg|}d} nZ|rtdd t|j d d |}fdd|D|}tdd|D| |\}}qt|dkrg}g}|D]Dfddt|Dd}|||d|||dq<|rtdd || d!|d"||}t||D]\|| | 7}q|dks| ||kr|rtd#d$|d%|| d | |7} || | <|D]Jd&d|Dkrlfd'dt|Dd}|||| | <q0q@|rtd|| dd$|d|| d(|d d }t||D]\}t| }t|dks|jj|krq|rtdd t|j d d |}fd)d|D}fd*dt|Dd}|fg|d||||dd}td+d|D| |\}}d,}qq|s@|rtd-| d7} |fg|}d} np|r,|dkrtd#d.|d/|| d n*td#d.|d/|| d0|| d | |7} || | <qt| S)1a Compute a device map for a given model giving priority to GPUs, then offload on CPU and finally offload to disk, such that: - we don't exceed the memory available of any of the GPU. - if offload to the CPU is needed, there is always room left on GPU 0 to put back the layer offloaded on CPU that has the largest size. - if offload to the CPU is needed,we don't exceed the RAM available on the CPU. - if offload to the disk is needed, there is always room left on the CPU to put back the layer offloaded on disk that has the largest size. All computation is done analyzing sizes and dtypes of the model parameters. As a result, the model can be on the meta device (as it would if initialized within the `init_empty_weights` context manager). Args: model (`torch.nn.Module`): The model to analyze. max_memory (`Dict`, *optional*): A dictionary device identifier to maximum memory. Will default to the maximum memory available if unset. no_split_module_classes (`List[str]`, *optional*): A list of layer class names that should never be split across device (for instance any layer that has a residual connection). dtype (`str` or `torch.dtype`, *optional*): If provided, the weights will be converted to that type when loaded. special_dtypes (`Dict[str, Union[str, torch.device]]`, *optional*): If provided, special dtypes to consider for some specific weights (will override dtype used as default for all weights). verbose (`bool`, *optional*, defaults to `False`): Whether or not to provide debugging statements as the function builds the device_map. NrcSsg|]}|dkr|qS))r^rr)rr@r)r)r*rsz)infer_auto_device_map..rrr^rrThe model weights are not tied. Please use the `tie_weights` method before using the `infer_auto_device` function.Frwz Treating module r.cs&g|]}|kr|ds|qSrrrrr)r*rscSs&g|]\}}t|tjjr||fqSr)r"r/rrrrmr)r)r*rscs<g|]4}tfdd|Drtfdd|Ds|qS)c3s|]}|kVqdSr|r)rrr)r*rsz3infer_auto_device_map...)rallrrrr)r*rsz' Found the relevant tied param groups csg|]}fdd|DqS)csg|]}|kr|qSr)r)rrr)r*rsz4infer_auto_device_map...r)rrr)r*rsz4 So those parameters need to be taken into account zNot enough space on z to put z (space available z, module size z).z6This module cannot be split, going to the next device.r z Splitting cs"g|]\}}d||fqSrr)rrr)r*rscSs&g|]\}}t|tjjr||fqSr)rrr)r)r*rscs g|]\}\}}|kr|qSr)r)rrrr) tied_paramr)r*r s z It looks like z is going to fit on z7 but we have tied parameters to account for. - Names z - Module names zPutting z and z on cSsg|] }|dqSrr))rrr)r)r*rscs g|]\}\}}|kr|qSr)r)rtied_module_namer)r*r s z, needed size cs"g|]\}}d||fqSrr)rrr)r*r8scs g|]\}\}}|kr|qSr)r)rrr)r*r9s cSs&g|]\}}t|tjjr||fqSr)rrr)r)r*rCsTz?None of the tied module can be split, going to the next device.z (size=z) on z (available=)rr"rrrQrMrNrrrrrrxrryrrprintrr ParameterrnrjrOzipr)rrrr-rrdevicesZgpusZ main_devicesrZtied_parametersrZcurrent_deviceZcurrent_memory_usedrZmax_layer_sizeZmax_layer_namesrWZ module_sizeZtied_param_goupsrr@Zcurrent_max_sizerZtied_module_namesZ tied_modulesZtied_module_indexZmodule_size_with_tiesZsplit_happenedZ tied_moduleZtied_module_childrenr))rrrr*infer_auto_device_mapqs"*      $     *     "$r)rrcstdd|D}|D],dkr8|qLqfdd|D}qt|dkrpd|}td|dS) z Checks a device map covers everything in a given model. Args: model (`torch.nn.Module`): The model to check the device map against. device_map (`Dict[str, Union[int, str, torch.device]]`): The device map to check. cSsg|] \}}|qSr)r)rrrr)r)r*risz$check_device_map..rcs&g|]}|ks|ds|qSrr)rrrr)r*rosrz, zOThe device_map provided does not give any device for the following parameters: N)rBrKrQclearrNrr')rrZall_model_tensorsZnon_covered_paramsr)rr*check_device_mapas    rc sT|dr.cs&g|]}|ks|dr|qSrrrrr)r*rsz#load_state_dict..cs"g|]}|tgkr|qSr))rrrdevice_weightsr)r*rsFcsg|]}t|qSr)rrr r)r*rsw)Zmain_process_onlytotalunitZ smoothingZleave)r r@)devrefresh)Z map_location)r%r ImportErrorrrIrQrrrOSErrorr'safe_load_filerNrrrrMrKextendrrrZ set_postfixset_descriptionZ get_tensorupdatecloser/loadr@) checkpoint_filerfrIZ weight_namesrr@Ztensors progress_barrRr))rrr*load_state_dict{sp             r) r checkpointrrr-offload_state_dictoffload_bufferskeep_in_fp32_modulesoffload_8bit_bnbc  s|rddlm} t|} t|r6t| dkr6tdt| ||dkrf|dk rfd|krft dn*|dk r|dk rd|krt j |dd t |t r|d d }tt|}d} d} t j|rt |d r|} n|g} nt j|rd dt |D} ddt |D}t| dkr:t j|| dg} nt|dkr^t j||dg} nnddt |D}t|dkrt |dtdtdn0t|dkrt j||d} nt |dnt d|d| dk rTt j| dt| d}t|}W5QRXd|kr.|d}ttt |} fdd| D} i}|rjt!"}i}dd|#D}| D]X}t$||d}|dkr|j$|ddn |%D]\}}d|krΐq|}t|dkr||krd|ddd}q|d kr(d |kr(t |d ||}|}|dk rt&|r|dk r|tj'krd}|D]2}||kr|d|ks||krhd}qqh|rtj(}d!|kr|d!d|)kr|j*tj+kr||d!d}nd}|dkrX|s||kr|dkr|j*}|r6| |||||||qnt,||d"|d#t-||||d$nr|d%kr|r|dkrx|j*}|r| |||||||n t,||d"|d#t-||||d$nt,||||||d&q~t./qt0|||rt1|||t23|t4|| dS)'a Loads a (potentially sharded) checkpoint inside a model, potentially sending weights to a given device as they are loaded. Once loaded across devices, you still need to call [`dispatch_model`] on your model to make it able to run. To group the checkpoint loading and dispatch in one single call, use [`load_checkpoint_and_dispatch`]. Args: model (`torch.nn.Module`): The model in which we want to load a checkpoint. checkpoint (`str` or `os.PathLike`): The folder checkpoint to load. It can be: - a path to a file containing a whole model state dict - a path to a `.json` file containing the index to a sharded checkpoint - a path to a folder containing a unique `.index.json` file and the shards of a checkpoint. - a path to a folder containing a unique pytorch_model.bin or a model.safetensors file. device_map (`Dict[str, Union[int, str, torch.device]]`, *optional*): A map that specifies where each submodule should go. It doesn't need to be refined to each parameter/buffer name, once a given module name is inside, every submodule of it will be sent to the same device. offload_folder (`str` or `os.PathLike`, *optional*): If the `device_map` contains any value `"disk"`, the folder where we will offload weights. dtype (`str` or `torch.dtype`, *optional*): If provided, the weights will be converted to that type when loaded. offload_state_dict (`bool`, *optional*, defaults to `False`): If `True`, will temporarily offload the CPU state dict on the hard drive to avoid getting out of CPU RAM if the weight of the CPU state dict + the biggest shard does not fit. offload_buffers (`bool`, *optional*, defaults to `False`): Whether or not to include the buffers in the weights offloaded to disk. keep_in_fp32_modules(`List[str]`, *optional*): A list of the modules that we keep in `torch.float32` dtype. offload_8bit_bnb (`bool`, *optional*): Whether or not to enable offload of 8-bit modules on cpu/disk. r )quantize_and_offload_8bitrrNrzeAt least one of the model submodule will be offloaded to disk, please pass along an `offload_folder`.T)exist_okrrz.jsoncSsg|]}|tkr|qSr))r rrr)r)r*r!sz,load_checkpoint_in_model..cSsg|]}|tkr|qSr))r r'r)r)r*r"scSsg|]}|dr|qS)z .index.json)r%r'r)r)r*r)s z6 is not a folder containing a `.index.json` file or a z or a z filezI containing more than one `.index.json` file, delete the irrelevant ones.z`checkpoint` should be the path to a file containing a whole state dict, or the index of a sharded checkpoint, or a folder containing a sharded checkpoint or the whole state dict, but got r.rrJcsg|]}tj|qSr))rrrr'Zcheckpoint_folderr)r*rBscSsg|] \}}|qSr)r)rr)r)r*rKs)rF)strictrarEz doesn't have any device set.rSr[r,)rUr^)rYr-rZ)5Zbnbr%rrrNrrrrr'rmakedirsr"r3rPrcr/risfiler%isdirlistdirrr r rbopenjsonloadsreadrrrtempfilemkdtempryrrKZis_floating_pointr;r9rQr-r<rtrgcZcollectrrshutilrmtreer)rr rrr-r!r"r#r$r%rZcheckpoint_filesZindex_filenameZpotential_state_binZpotential_state_safetensorZpotential_indexrrUZ offload_indexZstate_dict_folderZstate_dict_indexZ buffer_namesrrrqrZ param_deviceZ new_dtypeproceedrRrZr)r)r*load_checkpoint_in_models1              "          r9) native_ampautocast_kwargscCst}|dkri}n|}|r|jdkrFtjf|jjtjd|S|jdkr|jt j t j t j t j t jfkrtjf|jjtjd|Stjfd|jji|SntSdS)aH Return a context manager for autocasting mixed precision Args: native_amp (`bool`, *optional*, defaults to False): Whether mixed precision is actually enabled. cache_enabled (`bool`, *optional*, defaults to True): Whether the weight cache inside autocast should be enabled. NZfp16) device_typer-Zbf16r<)r Z to_kwargsZmixed_precisionr/Zautocastr@rir;Zdistributed_typerNOZ MULTI_CPUZ MULTI_GPUZ MULTI_NPUZ MULTI_XPUr: contextlib nullcontext)r:r;stater)r)r*#get_mixed_precision_context_managers"  rA)NNN)TF)NN)N)r)NNNNF)NNNNF)N)NNNFFNF)FN)Ur>r5rr0loggingrr1r6r3 collectionsrtypingrrrrrr/Ztorch.nnrr@r constantsr r dataclassesr rrZimportsrrrrZoffloadrrrrrZ torch_npuZ safetensorsrZsafetensors.torchrrZWEIGHTS_INDEX_NAME getLoggerrjrr#r3r+r-r5rkr@rArVrZ HalfTensorrtr0rzrr{rrrrrrrrrrrrrrrrPathLiker9rAr)r)r)r*s       ,%  V   :    ',5&     q& W  O