U ,-eE @sddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl m Z ddl mZddlmZmZddlmZmZmZmZmZmZmZddlZddlmZddlmZmZddlm Z m!Z!d d l"m#Z#d d l$m%Z%d d l&m'Z'd d l(m)Z)m*Z*d dl+m,Z,m-Z-m.Z.d dl/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6d dl7m8Z8m9Z9m:Z:m;Z;mZ>m?Z?m@Z@mAZAmBZBmCZCmDZDmEZEmFZFmGZGmHZHmIZImJZJmKZKmLZLmMZMmNZNmOZOmPZPmQZQmRZRmSZSmTZTmUZUmVZVmWZWd dlXmYZYmZZZd dl[m\Z\m]Z]m^Z^m_Z_d dl`maZambZbmcZcd dldmeZeejfgddhZiejfgddhZjeKrddlkmlZlmmZmmnZnddlompZpddlqmrZrmsZsmtZtmuZumvZvmwZwmxZxmyZyeSrddlzm{Z{ddl|m}Z~ddl|mZeUeZdadd Zd!d"Ze]rddlmmZdd#lmZeeed$kZnd%ZeQrd d&l7mZe dgd'd(Zeeje*d)fd*d+d,Zeeje*d)fd*d-d.Zeeje*d)fd*d/d0Zd1d2Zd3d4Zd5d6Zd7eBfeeejfeeefed8d9d:Zdhd;d<Zeeejfd=d>d?Zd@dAZdBdCZdDdEZdFdGZdidHdIZdjeeeedJdKdLZGdMd)d)ZGdNdOdOejee*eEe,ZeGeje_ejjdk rzejjjdPdQdRdSej_GdTdUdUejZGdVdWdWejZGdXdYdYejZeGdZd[d[eDZGd\d]d]ejZGd^d_d_ejZejejd`dadbZdcddZdedfZdS)kN)contextmanager) dataclass)partialwraps)AnyCallableDictListOptionalTupleUnion)version)Tensornn)CrossEntropyLossIdentity)get_activation)PretrainedConfig)custom_object_save)GenerationConfigGenerationMixin)PeftAdapterMixindeepspeed_configis_deepspeed_zero3_enabled)Conv1Dapply_chunking_to_forward find_pruneable_heads_and_indicesid_tensor_storageprune_conv1d_layer prune_layerprune_linear_layer) ADAPTER_SAFE_WEIGHTS_NAMEADAPTER_WEIGHTS_NAME CONFIG_NAME DUMMY_INPUTSFLAX_WEIGHTS_NAMESAFE_WEIGHTS_INDEX_NAMESAFE_WEIGHTS_NAMETF2_WEIGHTS_NAMETF_WEIGHTS_NAMEWEIGHTS_INDEX_NAME WEIGHTS_NAMEContextManagers ModelOutputPushToHubMixin cached_file copy_func download_urlextract_commit_hashhas_fileis_accelerate_availableis_auto_gptq_availableis_bitsandbytes_availableis_flash_attn_availableis_offline_modeis_optimum_availableis_peft_available is_remote_urlis_safetensors_availableis_torch_tpu_availableloggingreplace_return_docstrings strtobool)convert_file_size_to_intget_checkpoint_shard_files)ENV_VARS_TRUE_VALUESis_sagemaker_mp_enabledis_torch_fx_proxyis_torchdynamo_compiling)BitsAndBytesConfig GPTQConfigQuantizationMethod)require_version_core XLA_USE_BF160XLA_DOWNCAST_BF16)dispatch_modelinfer_auto_device_mapinit_empty_weights)add_hook_to_module)$check_tied_parameters_on_same_devicefind_tied_parametersget_balanced_memoryget_max_memoryload_offloaded_weightsoffload_weightsave_offload_indexset_module_tensor_to_device) safe_open) load_file) save_fileTcCs*tjo(tjo(ttjdddkS)NZACCELERATE_USE_FSDPFalser)torch distributed is_availableis_initializedrAosenvirongetrfrf\/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/modeling_utils.pyis_fsdp_enabled{s  rhcCstottjdddkS)NZ LOCAL_RANKr)rhintrcrdrerfrfrfrgis_fsdp_enabled_and_dist_rank_0srk __version__z1.10F)find_adapter_config_fileccs"t}|r daz dVW5|aXdS)z Context manager to globally disable weight initialization to speed up loading large models. TODO(Patrick): Delete safety argument `_enable=True` at next major version. . FN) _init_weights)_enableZold_init_weightsrfrfrgno_init_weightss  rqModuleUtilsMixin) parametercCsfzt|jWStk r`tjtttt fddd}|j |d}t|}|djYSXdS)NmodulereturncSsdd|jD}|S)NcSs"g|]\}}t|r||fqSrfr_Z is_tensor.0kvrfrfrg s zHget_parameter_device..find_tensor_attributes..__dict__itemsruZtuplesrfrfrgfind_tensor_attributessz4get_parameter_device..find_tensor_attributesZget_members_fnr) next parametersdevice StopIterationrModuler r strr_named_membersrsrgenZ first_tuplerfrfrgget_parameter_devices rcCsfzt|jWStk r`tjtttt fddd}|j |d}t|}|djYSXdS)z` Returns the first parameter dtype (can be non-floating) or asserts if none were found. rtcSsdd|jD}|S)NcSs"g|]\}}t|r||fqSrfrwrxrfrfrgr|s zMget_first_parameter_dtype..find_tensor_attributes..r}rrfrfrgrsz9get_first_parameter_dtype..find_tensor_attributesrrN) rrdtyperrrr r rrrrrfrfrgget_first_parameter_dtypes rcCsd}|D]n}|j}|r ttkr6tr6tjSttkrptrp|jtj krZtjS|jtj krptj S|jSq |dk r|St j ttttfddd}|j|d}d}|D]"}|}|dr|djSq|dk r|djS|D]}|j}|r|jSq|S)zz Returns the first found floating dtype in parameters if there is one, otherwise returns the last dtype it found. NrtcSsdd|jD}|S)NcSs"g|]\}}t|r||fqSrfrwrxrfrfrgr|s zGget_parameter_dtype..find_tensor_attributes..r}rrfrfrgrsz3get_parameter_dtype..find_tensor_attributesrr)rris_floating_pointrLrDr>r_bfloat16rNfloatdoublefloat32rrr r rrrbuffers)rsZ last_dtypetrrZ last_tupletuplerfrfrgget_parameter_dtypes8            rcCs,|D]}|r|jSqtddS)z_ Returns the first found floating dtype in `state_dict` or asserts if none were found. z5couldn't find any floating point dtypes in state_dictN)valuesrr ValueError state_dictrrfrfrgget_state_dict_float_dtypes  rcCs.|D]}|r|jSqt|jS)zt Returns the first found floating dtype in `state_dict` if there is one, otherwise returns the first dtype. N)rrrrrrfrfrgget_state_dict_dtypes  rcCsN|tjkrdStdt|}|dkr6td|dt|d}|dS)z Returns the size (in bytes) occupied by one parameter of type `dtype`. Example: ```py >>> dtype_byte_size(torch.float32) 4 ``` g?z [^\d](\d+)$Nz`dtype` is not a valid dtype: .r)r_boolresearchrrrjgroups)rZ bit_searchZbit_sizerfrfrgdtype_byte_sizes r10GB)rmax_shard_size weights_namec Cst|}ig}d}d}i}|D]\}}t|tr8q"nt|} | |kr^|| } ||| |<q"|t|j} || |krt|ddkr| id}||d|<|| 7}|| 7}t|d|| <q"t|dkr||didfSi} i} t |D]t\}}| dd|dddt|dd}| d d|dddt|dd }|| |<| D]}|| |<qXqd |i}|| d }| |fS) a Splits a model state dictionary in sub-checkpoints so that the final size of each sub-checkpoint does not exceed a given size. The sub-checkpoints are determined by iterating through the `state_dict` in the order of its keys, so there is no optimization made to make each sub-checkpoint as close as possible to the maximum size passed. For example, if the limit is 10GB and we have weights of sizes [6GB, 6GB, 2GB, 6GB, 2GB, 2GB] they will get sharded as [6GB], [6+2GB], [6+2+2GB] and not [6+2+2GB], [6+2GB], [6GB]. If one of the model's weight is bigger than `max_shard_size`, it will end up in its own sub-checkpoint which will have a size greater than `max_shard_size`. Args: state_dict (`Dict[str, torch.Tensor]`): The state dictionary of a model to save. max_shard_size (`int` or `str`, *optional*, defaults to `"10GB"`): The maximum size of each sub-checkpoint. If expressed as a string, needs to be digits followed by a unit (like `"5MB"`). weights_name (`str`, *optional*, defaults to `"pytorch_model.bin"`): The name of the model save file. rrirN.bin-Z05dz-of- .safetensors total_size)metadata weight_map) rBr isinstancerrnumelrrlenappend enumeratereplacekeys)rrrZsharded_state_dictsZlast_block_sizerZstorage_id_to_blockkeyweightZ storage_idZblock_idZ weight_sizershardsidxshard shard_filerindexrfrfrgshard_checkpoint"sJ     &  rc s"tj|t}tj|t}tj|}tj|}|sr|rBtsrtrPttfntf}tdd|d|dd} |r|rtrd} qt d|dn|sd} | r|n|} t | d d d } t | } W5QRXt t| d } | d |fd dD}fddD}|rt|dksHt|dkrd|jj}t|dkrddd|D}|d|d7}t|dkrddd|D}|d|d7}t|| rtn ttj dd}| D]0}|tj||}|j|dd~tqtjjj||S)a This is the same as [`torch.nn.Module.load_state_dict`](https://pytorch.org/docs/stable/generated/torch.nn.Module.html?highlight=load_state_dict#torch.nn.Module.load_state_dict) but for a sharded checkpoint. This load is performed efficiently: each checkpoint shard is loaded one by one in RAM and deleted after being loaded in the model. Args: model (`torch.nn.Module`): The model in which to load the checkpoint. folder (`str` or `os.PathLike`): A path to a folder containing the sharded checkpoint. strict (`bool`, *optional`, defaults to `True`): Whether to strictly enforce that the keys in the model state dict match the keys in the sharded checkpoint. prefer_safe (`bool`, *optional*, defaults to `False`) If both safetensors and PyTorch save files are present in checkpoint and `prefer_safe` is True, the safetensors files will be loaded. Otherwise, PyTorch files are always loaded when possible. Returns: `NamedTuple`: A named tuple with `missing_keys` and `unexpected_keys` fields - `missing_keys` is a list of str containing the missing keys - `unexpected_keys` is a list of str containing the unexpected keys zCan't find a checkpoint index ( or z) in rFTz"Cannot load sharded checkpoint at z+ safely since safetensors is not installed!rutf-8encodingrcsg|]}|kr|qSrfrfryr) loaded_keysrfrgr|sz+load_sharded_checkpoint..csg|]}|kr|qSrfrfr) model_keysrfrgr|sr#Error(s) in loading state_dict for ,cSsg|]}d|dqS"rfryrzrfrfrgr|sz Missing key(s): cSsg|]}d|dqSrrfrrfrfrgr|scpu map_location)strict) rcpathjoinr+r'isfiler=rloggerwarningopenjsonloadlistsetrrrr __class____name__ RuntimeErrorsafe_load_filerr_load_state_dictgccollectrmodulesruZ_IncompatibleKeys)modelfolderrZ prefer_safeZ index_fileZsafe_index_fileZ index_presentZsafe_index_present filenamesZ load_safeZ load_indexfrZ shard_files missing_keysunexpected_keys error_messageZstr_missing_keysZstr_unexpected_keysloaderrrrf)rrrgload_sharded_checkpointusT      " r)checkpoint_filec CsT|drxtrxt|dd}|}W5QRX|ddkrPtd|dn |ddkrptd|dd t|Sz>tst rt j rt j d krd }nd }t j||d WStk rN}zxzBt|0}|ddkrtdntd|d|W5QRXWn0ttfk r<td|d|dYnXW5d}~XYnXdS)z] Reads a PyTorch checkpoint file, returning properly formatted errors if they arise. rpt) frameworkformat)rtfZflaxz"The safetensors archive passed at zf does not contain the valid metadata. Make sure you save your model with the `save_pretrained` method.zConversion from a z7 safetensors archive to PyTorch is not implemented yet.rmetarrr zYou seem to have cloned a repository without having git-lfs installed. Please install git-lfs and run `git lfs install` followed by `git lfs pull` in the folder you cloned.zUnable to locate the file z_ which is necessary to load this pretrained model. Make sure you have saved the model properly.z9Unable to load weights from pytorch checkpoint file for 'z' at 'zZ'. If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.N)endswithr=r[rreOSErrorNotImplementedErrorrrrhr_r`rbget_rankr ExceptionrreadrUnicodeDecodeError)rrrrerfrfrgrsP     rcsN|D]@\}fdd|D}tt|t|dkrd|_qdS)z Sets the `_is_hf_initialized` flag in all submodules of a given model when all its weights are in the loaded state dict. cs.g|]&}|dr|ddqSr) startswithrr module_namerfrgr|sz.set_initialized_submodules..rTN) named_modulesrrrr_is_hf_initialized)rZstate_dict_keysrurrfrrgset_initialized_submoduless rcsg}g}|D]H}d}d|kr,|dd}d|kr@|dd}|r||||qt||D]\}}||||<qdt|dd|}dk r|_gd tj dfdd |||d ~S) Ngammarbetabias _metadatar)ruc s dkr inddi}||dggf}tfdd|Ddkrtrddl}t|jddddfdd|D}t|dkr|jj|dd t j dkr|j |W5QRXn |j ||j D]$\}}|dk r|||d qdS) NriTcsg|]}|r|qSrfrrprefixrfrgr|%s z=_load_state_dict_into_model..load..rF)rrecursecsg|]}|kr|qSrfrfr)named_parametersrfrgr|,sZ modifier_rankr)rerr deepspeeddictrrzeroGatheredParametersr_r`rZ_load_from_state_dict_modulesr) rurrZlocal_metadataargsr Zparams_to_gathernamechild error_msgsrr)rrrgr s   z)_load_state_dict_into_model..loadr)r) rrrzippopgetattrcopyrrr) model_to_loadr start_prefixold_keysnew_keysrnew_keyold_keyrfrrg_load_state_dict_into_models,      rcCst|dkr.||r.d|ddd}|d}|}t|dkrtt||drlt||d}|d=qsz3_load_state_dict_into_meta_model..rrvaluerrrirz doesn't have any device set.diskSCB)r0fp16_statistics) integrationsr+rrrrrrrrr_ranyfloat16r$rrinspect signaturerZrrrrrrrXint8)rrr&r expected_keys device_mapoffload_folder offload_indexstate_dict_folderstate_dict_indexr is_quantizedis_safetensorskeep_in_fp32_modulesr+rrrrrrparamrZset_module_kwargs old_paramsplitsrZ param_devicer3rfr.rg _load_state_dict_into_meta_modelts!                   $rF)rvariantrvcCs>|dk r:|d}|dd|g|dd}d|}|S)Nrri)rr)rrGrErfrfrg _add_variants   rHc@seZdZdZeddZeddZddZdd Ze e j d d d Z e e j d d dZ e e dddZed(ddZd)e eee j e je dddZd*ee eee dddZddZd+eeeddd Zeeee j effed!d"d#Zd,eeee j effeed%d&d'ZdS)-rrzH A few utilities for `torch.nn.Modules`, to be used as a mixin. cOsLz ddl}Wntk r(tdYnX|t}|}|j|_dS)NrFYou need to install psutil (pip install psutil) to use memory tracing.)psutil ImportErrorProcessrcgetpid memory_inforssmem_rss_pre_forward)rurkwargsrJprocessmemrfrfrg_hook_rss_memory_pre_forwards z-ModuleUtilsMixin._hook_rss_memory_pre_forwardcOsrz ddl}Wntk r(tdYnX|t}|}|j|_|j|j}|t |drf|j nd|_ dS)NrrI mem_rss_diff) rJrKrLrcrMrNrOmem_rss_post_forwardrPr rU)rurrQrJrRrSrUrfrfrg_hook_rss_memory_post_forwards  z.ModuleUtilsMixin._hook_rss_memory_post_forwardcCs2|D]}||j||jq|dS)a% Add a memory hook before and after each sub-module forward pass to record increase in memory consumption. Increase in memory consumption is stored in a `mem_rss_diff` attribute for each module and can be reset to zero with `model.reset_memory_hooks_state()`. N)rZregister_forward_pre_hookrTregister_forward_hookrWreset_memory_hooks_stateselfrurfrfrgadd_memory_hookss  z!ModuleUtilsMixin.add_memory_hookscCs$|D]}d|_d|_d|_qdS)z Reset the `mem_rss_diff` attribute of each module (see [`~modeling_utils.ModuleUtilsMixin.add_memory_hooks`]). rN)rrUrVrPrZrfrfrgrY(s z)ModuleUtilsMixin.reset_memory_hooks_statervcCst|S)z `torch.device`: The device on which the module is (assuming that all the module parameters are on the same device). )rr[rfrfrgr1szModuleUtilsMixin.devicecCst|S)zw `torch.dtype`: The dtype of the module (assuming that all the module parameters have the same dtype). )rr^rfrfrgr9szModuleUtilsMixin.dtype)encoder_attention_maskrvcCst|dkr(|dddddddf}|dkrL|ddddddf}|j|jd}d|t|jj}|S)z Invert an attention mask (e.g., switches 0. and 1.). Args: encoder_attention_mask (`torch.Tensor`): An attention mask. Returns: `torch.Tensor`: The inverted attention mask. Nr?)dimr$rr_finfomin)r[r_Zencoder_extended_attention_maskrfrfrginvert_attention_mask@s  z&ModuleUtilsMixin.invert_attention_maskNc Cs|dk rtdtn|j}|\}}tj||d}|ddddf||d|ddddfk}||j}|j d|j dkr|j d|j d}tj tj |||f||jd|gdd}|dddddddf|ddddddf}|S)NNThe `device` argument is deprecated and will be removed in v5 of Transformers.)rrrrri)Zaxis) warningswarn FutureWarningrr_Zarangerepeatr$rshapecatZones) input_shapeattention_maskrZ batch_sizeZ seq_lengthZseq_idsZ causal_maskZprefix_seq_lenextended_attention_maskrfrfrg*create_extended_attention_mask_for_decoderXs(. 4z;ModuleUtilsMixin.create_extended_attention_mask_for_decoder)rqrprrrvcCs|dkr|j}|dkr"|jjs6|dk r6tdt|dkr`|dddddddf}nV|dkr|jjrt|||}q|ddddddf}nt d|d|j d|j |d}d |t |j}|S) a Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (`torch.Tensor`): Mask with ones indicating tokens to attend to, zeros for tokens to ignore. input_shape (`Tuple[int]`): The shape of the input to the model. Returns: `torch.Tensor` The extended attention mask, with a the same dtype as `attention_mask.dtype`. Nrarhr`z!Wrong shape for input_ids (shape z) or attention_mask (shape )rbrc)rrdconfigZ is_decoderrjrkrlrrrsrrnr$r_rerf)r[rqrprrrrrfrfrgget_extended_attention_maskts0   z,ModuleUtilsMixin.get_extended_attention_maskF) head_masknum_hidden_layersis_attention_chunkedrvcCs6|dk r(|||}|dkr2|d}n dg|}|S)a Prepare the head mask if needed. Args: head_mask (`torch.Tensor` with shape `[num_heads]` or `[num_hidden_layers x num_heads]`, *optional*): The mask indicating if we should keep the heads or not (1.0 for keep, 0.0 for discard). num_hidden_layers (`int`): The number of hidden layers in the model. is_attention_chunked (`bool`, *optional*, defaults to `False`): Whether or not the attentions scores are computed by chunks or not. Returns: `torch.Tensor` with shape `[num_hidden_layers x batch x num_heads x seq_length x seq_length]` or list with `[None]` for each layer. NTri)_convert_head_mask_to_5d unsqueeze)r[rwrxryrfrfrg get_head_masks    zModuleUtilsMixin.get_head_maskcCs|dkr<|dddd}||dddd}n"|dkr^|ddd}|dks|td||j|jd}|S)zD-> [num_hidden_layers x batch x num_heads x seq_length x seq_length]rrrirazhead_mask.dim != 5, instead rb)rdr{expandAssertionErrorr$r)r[rwrxrfrfrgrzs  z)ModuleUtilsMixin._convert_head_mask_to_5d)only_trainableexclude_embeddingsrvcs|r.dd|Dfdd|D}n t|}g}t|dd}|rftr^ddl}ntd|D]B}|jsx|sj|rt ||j j r| | d qj| | qjt|S) a Get number of (optionally, trainable or non-embeddings) parameters in the module. Args: only_trainable (`bool`, *optional*, defaults to `False`): Whether or not to return only the number of trainable parameters exclude_embeddings (`bool`, *optional*, defaults to `False`): Whether or not to return only the number of non-embeddings parameters Returns: `int`: The number of parameters. cSs&g|]\}}t|tjr|dqS)z.weight)rr Embedding)ryr module_typerfrfrgr|s z3ModuleUtilsMixin.num_parameters..csg|]\}}|kr|qSrfrf)ryrrsZembedding_param_namesrfrgr|sis_loaded_in_4bitFrNzbitsandbytes is not installed but it seems that the model has been loaded in 4bit precision, something went wrong make sure to install bitsandbytes with `pip install bitsandbytes`.ra)rrrrrr7 bitsandbytesrZ requires_gradrrZ Params4bitrrsum)r[rrZtotal_parametersZ total_numelrZbnbrCrfrrgnum_parameterss,     zModuleUtilsMixin.num_parameters) input_dictrvcCsJt|dsi|_|j|kr(||jSd|jkrFtdd|jd<dS)z Helper function to estimate the total number of tokens from the model inputs. Args: inputs (`dict`): The model inputs. Returns: `int`: The total number of tokens. warnings_issuedestimate_tokenszdCould not estimate the number of tokens of the input, floating-point operations will not be computedTr)r rmain_input_namerrr)r[rrfrfrgrs    z ModuleUtilsMixin.estimate_tokensT)rrrvcCsd|||j|dS)a Get number of (optionally, non-embeddings) floating-point operations for the forward and backward passes of a batch with this transformer model. Default approximation neglects the quadratic dependency on the number of tokens (valid if `12 * d_model << sequence_length`) as laid out in [this paper](https://arxiv.org/pdf/2001.08361.pdf) section 2.1. Should be overridden for transformers with parameter re-use e.g. Albert or Universal Transformers, or if doing long-range modeling with very high sequence lengths. Args: batch_size (`int`): The batch size for the forward pass. sequence_length (`int`): The number of tokens in each line of the batch. exclude_embeddings (`bool`, *optional*, defaults to `True`): Whether or not to count embedding and softmax operations. Returns: `int`: The number of floating-point operations. )r)rr)r[rrrfrfrgfloating_point_opssz#ModuleUtilsMixin.floating_point_ops)N)NN)F)FF)T)r __module__ __qualname____doc__ staticmethodrTrWr\rYpropertyr_rrrrgrsr rjrrvr rr|rzrrrr rrrrfrfrfrgrrsN      5  /"c seZdZdZdZdZdZdZdZdZ dZ dZ dZ dZ dZdZdZdZeeeejfdddZeedd d Zed fd d ZddZddZeddZeejejdddZ ee!j"dddZ#ee$dddZ%edte&eje&e'eeee(ffedddZ)ddZ*d d!Z+e!j"dd"d#Z,e!j"d$d%d&Z-e!j"dd'd(Z.d)d*Z/d+d,Z0d-d.Z1e2e!j"e!j"ed/d0d1Z3d2d3Z4due&e(e&e(e!j5d4d5d6Z6dvd7d8Z7dwe!j5e&e(e&e(e!j5d9d:d;Z8dxe!j9e&e(e&e$e!j9d<d=d>Z:d?d@Z;e(dAdBdCZdFdGZ?ee(e@e(fdHdIdJZAdKdLZBdMdNZCee$ddOdPZDdQdejEddRddddQf e'eeFjGfe$e&eHeIe$e'e(efe$e&ee&e'ee$fe$dS dTdUZJdydVdWZKeLej!j"jMfdXdYZMeLej!j"jNfdZd[ZNfd\d]ZOfd^d_ZPeddddddd`ddae&e'eeFjGfe&e'eeeFjGfe&e'eeFjGfe$e$e$e&e'ee$fee$db dcddZQedzdedfZRd{dgdhZSe2d|didjZTed}dldmZUdddndoZVdpdqZWdrdsZXZYS)~PreTrainedModela Base class for all models. [`PreTrainedModel`] takes care of storing the configuration of the models and handles methods for loading, downloading and saving models as well as a few methods common to all models to: - resize the input embeddings, - prune heads in the self-attention heads. Class attributes (overridden by derived classes): - **config_class** ([`PretrainedConfig`]) -- A subclass of [`PretrainedConfig`] to use as configuration class for this model architecture. - **load_tf_weights** (`Callable`) -- A python *method* for loading a TensorFlow checkpoint in a PyTorch model, taking as arguments: - **model** ([`PreTrainedModel`]) -- An instance of the model on which to load the TensorFlow checkpoint. - **config** ([`PreTrainedConfig`]) -- An instance of the configuration associated to the model. - **path** (`str`) -- A path to the TensorFlow checkpoint. - **base_model_prefix** (`str`) -- A string indicating the attribute associated to the base model in derived classes of the same architecture adding modules on top of the base model. - **is_parallelizable** (`bool`) -- A flag indicating whether this model supports model parallelization. - **main_input_name** (`str`) -- The name of the principal input to the model (often `input_ids` for NLP models, `pixel_values` for vision models and `input_values` for speech models). Nr input_idsFr]cCsdttiS)z^ `Dict[str, torch.Tensor]`: Dummy inputs to do a forward pass in the network. r)r_tensorr%r^rfrfrg dummy_inputscszPreTrainedModel.dummy_inputscCsdS)z@ :str: Identifies that this is a PyTorch model. rrfr^rfrfrgrjszPreTrainedModel.frameworkrucsbtt|ts2td|jjd|jjd||_|j|_i|_ | rXt |nd|_ dS)NzParameter config in `zt(config)` should be an instance of class `PretrainedConfig`. To create a model from a pretrained model use `model = z(.from_pretrained(PRETRAINED_MODEL_NAME)`)super__init__rrrrrru name_or_pathr can_generaterZfrom_model_configgeneration_config)r[ruinputsrQrrfrgrqs  zPreTrainedModel.__init__cCs||dS)z A method executed at the end of each Transformer model initialization, to execute code that needs the model's modules properly initialized (such as weight initialization). N) init_weights._backward_compatibility_gradient_checkpointingr^rfrfrg post_initszPreTrainedModel.post_initcCs,|jr(t|jddr(|t|jddS)Ngradient_checkpointingF)supports_gradient_checkpointingrrugradient_checkpointing_enabledelattrr^rfrfrgrsz>PreTrainedModel._backward_compatibility_gradient_checkpointingc Ks|dd}d}|dk r"||}trdddl}td|jjtd||f|}W5QRXn ||f|}|dk rt ||S)z All context managers that the model should be initialized under go here. Args: torch_dtype (`torch.dtype`, *optional*): Override the default `torch.dtype` and load the model under this dtype. torch_dtypeNr@Detected DeepSpeed ZeRO-3: activating zero.init() for this modelZconfig_dict_or_path) r_set_default_torch_dtyperr rinfor Initrr_set_default_dtype)clsrurQr dtype_origr rrfrfrg _from_configs     zPreTrainedModel._from_config)rrvcCsN|jstd|jd|dtd|jd|dt}t||S)a Change the default dtype and return the previous one. This is needed when wanting to instantiate the model under specific dtype. Args: dtype (`torch.dtype`): a floating dtype to set to. Returns: `torch.dtype`: the original `dtype` that can be used to restore `torch.set_default_dtype(dtype)` if it was modified. If it wasn't, returns `None`. Note `set_default_dtype` currently only works with floating-point types and asserts if for example, `torch.int64` is passed. So if a non-float `dtype` is passed this functions will throw an exception. zCan't instantiate z model under dtype=z' since it is not a floating point dtypezInstantiating z model under default dtype r)rrrrrr_Zget_default_dtyper)rrrrfrfrgrs z(PreTrainedModel._set_default_torch_dtypecCst||j|S)z@ `torch.nn.Module`: The main body of the model. )rbase_model_prefixr^rfrfrg base_modelszPreTrainedModel.base_modelcCs$dt|jkr dt|jkr dSdS)z Returns whether this model can generate sequences with `.generate()`. Returns: `bool`: Whether this model can generate sequences with `.generate()`. rFT)rZprepare_inputs_for_generationgeneraterrfrfrgrs zPreTrainedModel.can_generate)rr;rvcCs|jstdtstdn2ttjd}|tdk}|sPtd|t|dd}|rhtd|d kr|t d n(|d k r|t j t j fkrtd |d |d krt jrt d qtdn2|d k rt|trd|ksd|krtdd|_|S)a If you don't know about Flash Attention, check out the official repository of flash attention: https://github.com/Dao-AILab/flash-attention For using Flash Attention 1.0 you can do it directly via the `BetterTransformer` API, have a look at this specific section of the documentation to learn more about it: https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#decoder-models The method checks if the current setup is compatible with Flash Attention as it requires the model to be in half precision and not ran on CPU. If all checks pass, the method will create an attribute in the config `_flash_attn_2_enabled` so that the model can initialize the correct attention module zThe current architecture does not support Flash Attention 2.0. Please open an issue on GitHub to request support for this architecture: https://github.com/huggingface/transformers/issues/newzFlash Attention 2.0 is not available. Please refer to the documentation of https://github.com/Dao-AILab/flash-attention for installing it.Z flash_attnz2.0.0zxYou need flash_attn package version to be greater than 2.0. Make sure to have that version installed - detected version Zuse_bettertransformerFzFlash Attention 2 and BetterTransformer API are not compatible. Please make sure to disable BetterTransformers by doing model.reverse_bettertransformer()NzwYou are attempting to use Flash Attention 2.0 without specifying a torch dtype. This might lead to unexpected behaviourzVFlash Attention 2.0 only supports torch.float16 and torch.bfloat16 dtypes. You passed z*, this might lead to unexpected behaviour.zYou are attempting to use Flash Attention 2.0 with a model initialized on CPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.a)You are attempting to use Flash Attention 2.0 with a model initialized on CPU and with no GPU available. This is not supported yet. Please make sure to have access to a GPU and either initialise the model on a GPU by passing a device_map or initialising the model on CPU and then moving it to GPU.rr1zYou are attempting to use Flash Attention 2.0 with a model dispatched on CPU or disk. This is not supported. Please make sure to initialise the model on a GPU by passing a device_map that contains only GPU devices as keys.T)_supports_flash_attn_2rr8rKr parse importlibrrrrr_r6rcudararr rZ_flash_attn_2_enabled)rrurr;Zflash_attention_versionZis_flash_greater_than_2Z_is_bettertransformerrfrfrg_check_and_enable_flash_attn_2s`     z.PreTrainedModel._check_and_enable_flash_attn_2cCsdd}|||_dS)z Enables the gradients for the input embeddings. This is useful for fine-tuning adapter weights while keeping the model weights fixed. cSs|ddS)NT)Zrequires_grad_)ruinputoutputrfrfrgmake_inputs_require_grads2szMPreTrainedModel.enable_input_require_grads..make_inputs_require_gradsN)get_input_embeddingsrX_require_grads_hook)r[rrfrfrgenable_input_require_grads,sz*PreTrainedModel.enable_input_require_gradscCs|jdS)z4 Removes the `_require_grads_hook`. N)rremover^rfrfrgdisable_input_require_grads7sz+PreTrainedModel.disable_input_require_gradscCs&t||j|}||k r|StdS)z Returns the model's input embeddings. Returns: `nn.Module`: A torch module mapping vocabulary to hidden states. N)rrrr)r[rrfrfrgr=sz$PreTrainedModel.get_input_embeddingsr0cCs*t||j|}||k r"||ntdS)z Set model's input embeddings. Args: value (`nn.Module`): A module mapping vocabulary to hidden states. N)rrset_input_embeddingsr)r[r0rrfrfrgrJs z$PreTrainedModel.set_input_embeddingscCsdS)z Returns the model's output embeddings. Returns: `nn.Module`: A torch module mapping hidden states to vocabulary. Nrfr^rfrfrgget_output_embeddingsWsz%PreTrainedModel.get_output_embeddingscCsdS)z\ Initialize the weights. This method should be overridden by derived class. NrfrZrfrfrgro`szPreTrainedModel._init_weightscCs$t|ddrdS||d|_dS)zM Initialize the weights if they are not already initialized. rFNT)rrorrZrfrfrg_initialize_weightsfs  z#PreTrainedModel._initialize_weightscCst|jddr.|}|dk r.|||t|jddrvt|jddrvt||jrbt||j}||j|j |j| D]}t|dr~| q~dS)z Tie the weights between the input embeddings and the output embeddings. If the `torchscript` flag is set in the configuration, can't handle parameter sharing so we are cloning the weights instead. tie_word_embeddingsTNZis_encoder_decoderFZtie_encoder_decoder _tie_weights) rrur_tie_or_clone_weightsrr r_tie_encoder_decoder_weightsencoderdecoderrr)r[output_embeddingsrurfrfrg tie_weightsos    zPreTrainedModel.tie_weights)rrrcszg}|j|jkr*t|jd|jddtjtjtttdfdd ||||t|dkrvtd|dS) N and zZ are not equal. In this case make sure that all encoder weights are correctly initialized.r)decoder_pointerencoder_pointerruninitialized_encoder_weightsc st|tjrt|tjs,t|d|dt|drpt|dsDt|j|_t|drlt|dsdt|j|_dS|j}|j}t|dkrt|dkstd|d|fdd | D}d}| D]\} } | r(t t | |} | } t|| t|| sRt|t|krR|d 8}qn*| |kr6qn|d krJtd n| } } || || d | ||d d|d | q|t|7}dS)Nrz have to be of type nn.ModulerrrzEncoder module z does not match decoder module csh|]}d|qS)/rf)ryZsub_namerrfrg szkPreTrainedModel._tie_encoder_decoder_weights..tie_encoder_to_decoder_recursively..rizMax depth of recursive function `tie_encoder_to_decoder` reached. It seems that there is a circular dependency between two or more `nn.Modules` of your model.r)depth)rrrrr rrrrrrisdigitrrjtyperrr) rrrrrZencoder_modulesZdecoder_modulesZall_encoder_weightsZencoder_layer_posrruZ encoder_nameZ decoder_name"tie_encoder_to_decoder_recursivelyrrgrsb       zXPreTrainedModel._tie_encoder_decoder_weights..tie_encoder_to_decoder_recursivelyz;The following encoder weights were not tied to the decoder )r) rrrrrrr rr)rrrrrfrrgrs"  < z,PreTrainedModel._tie_encoder_decoder_weightscCs|jjrt|j|_n|j|_t|dddk rftj|j j d|jj d|j j dfdd|j _ t |drt |dr|j |_dS)zPTie or clone module weights depending of whether we are using TorchScript or notrNrZconstant out_featuresnum_embeddings)ruZ torchscriptrr#rcloner functionalpadrdatarnr rr)r[rZinput_embeddingsrfrfrgrs z%PreTrainedModel._tie_or_clone_weights)new_num_tokenspad_to_multiple_ofrvcCsJ|||}|dkr |dkr |S|jjd|j_|jjd|_||S)a Resizes input token embeddings matrix of the model if `new_num_tokens != config.vocab_size`. Takes care of tying weights embeddings afterwards if the model class has a `tie_weights()` method. Arguments: new_num_tokens (`int`, *optional*): The number of new tokens in the embedding matrix. Increasing the size will add newly initialized vectors at the end. Reducing the size will remove vectors from the end. If not provided or `None`, just returns a pointer to the input tokens `torch.nn.Embedding` module of the model without doing anything. pad_to_multiple_of (`int`, *optional*): If set will pad the embedding matrix to a multiple of the provided value.If `new_num_tokens` is set to `None` will just pad the embedding to a multiple of `pad_to_multiple_of`. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta), or on TPUs which benefit from having sequence lengths be a multiple of 128. For more details about this, or help on choosing the correct value for resizing, refer to this guide: https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tc Return: `torch.nn.Embedding`: Pointer to the input tokens Embeddings Module of the model. Nr)_resize_token_embeddingsrrnruZ vocab_sizer)r[rrZ model_embedsrfrfrgresize_token_embeddingss z'PreTrainedModel.resize_token_embeddingsc Cs|}||||}t|dr0|j}t|||||dk rtr|ddl}|jj |j dd|j j d}W5QRXn |j j d}| dk r|j js| }|||}t|dr|j}t|||||S)N_hf_hookrr )r_get_resized_embeddingsr rrRrrr r r rrnrrur_get_resized_lm_headZset_output_embeddings) r[rrold_embeddingsnew_embeddingshookr  old_lm_head new_lm_headrfrfrgr s(        z(PreTrainedModel._resize_token_embeddings)rrrrvc Cs|dk rLt|ts"td|d|dkr6|jjd}||d||}ntd|d|dkrj|Strddl}|j j |jdd|j \}}W5QRXn|j \}}||krts|St|t j std t|d t j d t j d t j |||jj|jjd }||t||}trddl}|j|jg} |j j | dd2|jjd|ddf|jjd|ddf<W5QRXn,|jjd|ddf|jjd|ddf<|S)aj Build a resized Embedding Module from a provided token Embedding Module. Increasing the size will add newly initialized vectors at the end. Reducing the size will remove vectors from the end Args: old_embeddings (`torch.nn.Embedding`): Old embeddings to be resized. new_num_tokens (`int`, *optional*): New number of tokens in the embedding matrix. Increasing the size will add newly initialized vectors at the end. Reducing the size will remove vectors from the end. If not provided or `None`, just returns a pointer to the input tokens `torch.nn.Embedding` module of the model without doing anything. pad_to_multiple_of (`int`, *optional*): If set will pad the embedding matrix to a multiple of the provided value. If `new_num_tokens` is set to `None` will just pad the embedding to a multiple of `pad_to_multiple_of`. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta), or on TPUs which benefit from having sequence lengths be a multiple of 128. For more details about this, or help on choosing the correct value for resizing, refer to this guide: https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tc Return: `torch.nn.Embedding`: Pointer to the resized Embedding Module or the old Embedding Module if `new_num_tokens` is `None` Nz5Asking to pad the embedding matrix to a multiple of `z@`, which is not and integer. Please make sure to pass an integerrrzYou are resizing the embedding layer without providing a `pad_to_multiple_of` parameter. This means that the new embedding dimension will be a.. This might induce some performance reduction as *Tensor Cores* will not be available. For more details about this, or help on choosing the correct value for resizing, refer to this guide: https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tcr zOld embeddings are of type , which is not an instance of zj. You should either use a different resize function or make sure that `old_embeddings` are an instance of rri)rrjrrrnrrrr r r sizerr TypeErrorrrrrorfr) r[rrrr old_num_tokensZold_embedding_dimrnparamsrfrfrgr&sN"         8,z'PreTrainedModel._get_resized_embeddings)rr transposedrvc Cst|dkr |StrZddl}|jj|jdd&|s<|jn |j\}}W5QRXn |sh|jn |j\}}||krts|St|tj st dt |dtj dtj d|s||fn||f}|j dk }tj |||jj |jjd} || t||} tr^ddl}|j|j | j| j g} |jj| dd|| || ||W5QRXn|| || ||| S) aN Build a resized Linear Module from a provided old Linear Module. Increasing the size will add newly initialized vectors at the end. Reducing the size will remove vectors from the end Args: old_lm_head (`torch.nn.Linear`): Old lm head liner layer to be resized. new_num_tokens (`int`, *optional*): New number of tokens in the linear matrix. Increasing the size will add newly initialized vectors at the end. Reducing the size will remove vectors from the end. If not provided or `None`, just returns a pointer to the input tokens `torch.nn.Linear` module of the model without doing anything. transposed (`bool`, *optional*, defaults to `False`): Whether `old_lm_head` is transposed or not. If True `old_lm_head.size()` is `lm_head_dim, vocab_size` else `vocab_size, lm_head_dim`. Return: `torch.nn.Linear`: Pointer to the resized Linear Module or the old Linear Module if `new_num_tokens` is `None` Nrr z#Old language model head is of type rzg. You should either use a different resize function or make sure that `old_lm_head` are an instance of r)rrr)rr r r rrrrrLinearrrrrrrorf!_copy_lm_head_original_to_resized) r[rrrr rZold_lm_head_dimZnew_lm_head_shapehas_new_lm_head_biasrnum_tokens_to_copyrrfrfrgrsX    z$PreTrainedModel._get_resized_lm_headcCs|s2|jjd|ddf|jjd|ddf<n,|jjddd|f|jjddd|f<|r~|jjd||jjd|<dSr,)rrr)r[rrrrrrfrfrgrs .,z1PreTrainedModel._copy_lm_head_original_to_resized)new_num_position_embeddingscCs(td|jd|jd|jjddS)Nz4`resize_position_embeddings` is not implemented for B`. To implement it, you should overwrite this method in the class  in `modeling_.py`rrr)r[rrfrfrgresize_position_embeddingssz*PreTrainedModel.resize_position_embeddingscCs(td|jd|jd|jjddS)Nz1`get_position_embeddings` is not implemented for rrrrr^rfrfrgget_position_embeddingssz'PreTrainedModel.get_position_embeddingscCs2|jjr||jjtr.||j|dS)z If needed prunes and maybe initializes weights. If using a custom `PreTrainedModel`, you need to implement any initialization logic in `_init_weights`. N)ru pruned_heads prune_headsroapplyrrr^rfrfrgrs  zPreTrainedModel.init_weights)heads_to_prunecCsN|D]4\}}t|jj|gt|B}t||jj|<q|j|dS)a Prunes heads of the base model. Arguments: heads_to_prune (`Dict[int, List[int]]`): Dictionary with keys being selected layer indices (`int`) and associated values being the list of heads to prune in said layer (list of `int`). For instance {1: [0, 2], 2: [2, 3]} will prune heads 0 and 2 on layer 1 and heads 2 and 3 on layer 2. N)rrrurrerrZ _prune_heads)r[rlayerZheadsZ union_headsrfrfrgr s zPreTrainedModel.prune_headscCsD|jst|jjd|t|jddt|ddr@|dS)z Activates gradient checkpointing for the current model. Note that in other frameworks this feature can be referred to as "activation checkpointing" or "checkpoint activations". z) does not support gradient checkpointing.Tr_hf_peft_config_loadedFN) rrrrrr_set_gradient_checkpointingrrr^rfrfrgrs  z-PreTrainedModel.gradient_checkpointing_enablecCs2|jr|t|jddt|ddr.|dS)z Deactivates gradient checkpointing for the current model. Note that in other frameworks this feature can be referred to as "activation checkpointing" or "checkpoint activations". FrrN)rrrrrrr^rfrfrggradient_checkpointing_disable-s z.PreTrainedModel.gradient_checkpointing_disablecCstdd|DS)z Whether gradient checkpointing is activated for this model or not. Note that in other frameworks this feature can be referred to as "activation checkpointing" or "checkpoint activations". css|]}t|do|jVqdS)rN)r rrymrfrfrgr/Bsz.)r5rr^rfrfrgis_gradient_checkpointing:sz)PreTrainedModel.is_gradient_checkpointingTr) save_directoryis_main_processr save_function push_to_hubrsafe_serializationrGtokensave_peft_formatc 0 s| dd} | dk r4tdt| dk r0td| } | dk rD| | d<t|dd} t|ddrtt|d dst| sttd t|d dr| std d | krtd| d }|rtstdt j |rt d|ddSt j|dd|r.| dd}| d|t j jd}|j|f| }||}t|}t|}t|dd|j_|jjg|j_|jdk r~t|||jd|r8| s|j||r|j|| r8t d|!}| rt di}|"D]\}}||d|<q|}|#}t$|dkrtd|d}|j%|}|||dkrJ|&}t'rnt(j)j*j+D]\}}||}qZ|j,dk r|j,D]}||-kr||=q|rt./t0}|"D]\}|t1|2qdd |"D}t3}|4D]}|j5dk rZd} t6|D]Jt7fd!d"|j5D}!|!r|kr| d7} | t$|kr|=qd} |D]2|krb| d7} | dkrb|=|8qbqt$|dkrt 9d#|d$| s|rt:nt;}"t<|"|}"n|rt=nt>}"t?|||"d%\}#}$t @|D]}%t j A||%}&|"Bd&d'Bd(d'}'|%Bd&d'Bd(d'}(tCDd)})|%E|'rt j |&r|%|#-kr|r|)F|(dk rt G|&q|#"D]B\}*}+|rtH|+t j A||*d*d+id,n||+t j A||*q|$dkrt j A|t If a single weight of the model is bigger than `max_shard_size`, it will be in its own checkpoint shard which will be bigger than `max_shard_size`. safe_serialization (`bool`, *optional*, defaults to `False`): Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`). variant (`str`, *optional*): If specified, weights are saved in the format pytorch_model..bin. token (`str` or `bool`, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, or not specified, will use the token generated when running `huggingface-cli login` (stored in `~/.huggingface`). save_peft_format (`bool`, *optional*, defaults to `True`): For backward compatibility with PEFT library, in case adapter weights are attached to the model, all keys of the state dict of adapters needs to be pre-pended with `base_model.model`. Advanced users can disable this behaviours by setting `save_peft_format` to `False`. kwargs (`Dict[str, Any]`, *optional*): Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method. use_auth_tokenNVThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers.V`token` and `use_auth_token` are both specified. Please set only the argument `token`.r rFis_loaded_in_8bitis_8bit_serializablezYou are calling `save_pretrained` to a 8-bit converted model you may likely encounter unexepected behaviors. If you want to save 8-bit models, make sure to have `bitsandbytes>0.37.2` installed.rz]You are calling `save_pretrained` on a 4-bit converted model. This is currently not supportedZ save_configze`save_config` is deprecated and will be removed in v5 of Transformers. Use `is_main_process` instead.zR`safe_serialization` requires the `safetensors library: `pip install safetensors`.zProvided path (z#) should be a directory, not a fileTexist_okcommit_messagerepo_idrirrrzhDetected adapters on the model, saving the model in the PEFT format, only adapter weights will be saved.zTo match the expected format of the PEFT library, all keys of the state dict of adapters will be pre-pended with `base_model.model`.zbase_model.model.zMultiple active adapters detected, saving multiple active adapters is not supported yet. You can save adapters separately one by one by iteratively calling `model.set_adapter(adapter_name)` then `model.save_pretrained(...)`rcSs"i|]\}}t|dkr||qSrr)ryZptrnamesrfrfrg s z3PreTrainedModel.save_pretrained..c3s|]}t|VqdSr,rr)rypatrrfrgr/sz2PreTrainedModel.save_pretrained..zRemoved shared tensor zk while saving. This should be OK, but check by verifying that you don't receive any warning while reloading)rrrrrz(.*?)-\d{5}-of-\d{5}rr)rzModel weights saved in wrrra)indent sort_keys z:The model is bigger than the maximum size per checkpoint (z) and is going to be split in z^ checkpoint shards. You can find where each parameters has been saved in the index located at )rr )Prrjrkrlrrrr=rKrcrrrerrormakedirsrsepZ _create_repoZ_get_files_timestamps unwrap_modelrrrurrr architectures _auto_classrsave_pretrainedrrrZget_adapter_state_dictrZactive_adaptersrZ peft_configrIS_SAGEMAKER_MP_POST_1_10smpstateZmodule_managerZtranslate_functions_keys_to_ignore_on_saver collections defaultdictrrrrr_tied_weights_keyssortedr5add warning_oncer(r,rHr"r#rlistdirrrrcompiler fullmatchrsafe_save_filer'r+rrdumpswriteZ_upload_modified_files)0r[rr rr r rr rGr rrQrrrrZfiles_timestampsZ model_to_saverZpeft_state_dictrr0Zactive_adapterZcurrent_peft_configZ smp_to_hf_Z ignore_keyptrsrZ shared_ptrsZ warn_namesrfoundZmatches_patternrrrfilenameZ full_filenameZweights_no_suffixZfilename_no_suffixregrrZpath_to_weightsZsave_index_filercontentrfrrgr)Ds4=                                  zPreTrainedModel.save_pretrainedcCs<tdd|D}|r8tdd|D}||}|S)a Get the memory footprint of a model. This will return the memory footprint of the current model in bytes. Useful to benchmark the memory footprint of the current model and design some tests. Solution inspired from the PyTorch discussions: https://discuss.pytorch.org/t/gpu-memory-that-model-uses/56822/2 Arguments: return_buffers (`bool`, *optional*, defaults to `True`): Whether to return the size of the buffer tensors in the computation of the memory footprint. Buffers are tensors that do not require gradients and not registered as parameters. E.g. mean and std in batch norm layers. Please see: https://discuss.pytorch.org/t/what-pytorch-means-by-buffers/120266/2 cSsg|]}||qSrfZnelementZ element_size)ryrCrfrfrgr|isz8PreTrainedModel.get_memory_footprint..cSsg|]}||qSrfr@)rybufrfrfrgr|ks)rrr)r[Zreturn_buffersrSZmem_bufsrfrfrgget_memory_footprint]s z$PreTrainedModel.get_memory_footprintcs.t|ddtjkrtdntj||SdS)Nquantization_methodzCalling `cuda()` is not supported for `4-bit` or `8-bit` quantized models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct `dtype`.)rrJBITS_AND_BYTESrrrr[rrQrrfrgros zPreTrainedModel.cudacs.t|ddtjkrtdntj||SdS)NrCz`.to` is not supported for `4-bit` or `8-bit` bitsandbytes models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct `dtype`.)rrJrDrrr$rErrfrgr$zs zPreTrainedModel.tocs&t|ddrtdn tj|SdS)Nr@Fz`.half()` is not supported for quantized model. Please use the model as it is, since the model has already been casted to the correct `dtype`.)rrrhalfr[rrrfrgrFs  zPreTrainedModel.halfcs&t|ddrtdn tj|SdS)Nr@Fz`.float()` is not supported for quantized model. Please use the model as it is, since the model has already been casted to the correct `dtype`.)rrrrrGrrfrgrs  zPreTrainedModel.floatmain)ru cache_dirignore_mismatched_sizesforce_downloadlocal_files_onlyr revisionuse_safetensors) pretrained_model_name_or_pathrurIrJrKrLr rMrNcYs| dd} | dd} | dd}| dd}| dd}| dd}| d d}| d d}| d d}| d d}| d d}| dd}| dd| dd}| dd| dd}| dd}| dd}| dd}| dd}| dd}| dd}| dd} | dd}!| di}"| dd}#| d d}$trPd}|dk r|td!t|dk rxtd"|}|dk r|"dk rd#|"kr||"d#<| dkrtsd} trt t j d$t d%k}%nd}%|dkrt d&| dkrDt|ts8t|t||||||||ddd' }&t|&| } n t|dd} tr|"d(d}'|'dkrt|f|||||| d)|"}'|'dk rtj|'rt|'d*d+d,}(|}'t|(d-}W5QRXnd}'ttjrdinzttr4d.kr4zdtiWn&tk r0td/d0YnXn(tt r\d1krTtd2ndidk r|dkrvd}n|std3|rdk rt!d4t"rtd5nt#st$d6d})|dk rt|d7t%j&})|dkr|s|rt%j&})t'j(f||d8dd9| \}} n@|)t%j&krX|j)}|j*}d:d;| +D}*t,|*d1krXtd<|sd|rt#rtts|t$d=dkrt -d>d?tj.dkrtj/0rdtj/1intd@t -dA|dkrd}| s|rtdB| |B }+dCdD|dE},|dk r ||,dF<t2r<|s(|j6}||_6dkrtj.n t -dP|4(|?}/|%r|)t%j&kr|r|0t%j&krt dQ||_6n|%r|s|0t%j&kr|j6}t|t@r2t'j(|ddR}n"t|t'r@ntdStA|dT|j)}|rڈdkrptj.dkrtj/0rdtj/1intd@t -dA|dkrd}n"|%s|s|0t%j&krt dUd}5d}6d}7dd}8|dk rt|}tjB|}9|9 r| rNtjtjC||tDdVrNtjC||tDdV}: q| rtjtjC||tErtjC||tE}: q|rtjtjC||tFrtjC||tF}: q| dk rtjtjC||tGtH|!rtjC||tGtH|!}: q| dk rBtjtjC||tGtI|! rBtjC||tGtI|!}:d}5 qtjtjC||tGtJ|! r|tjC||tGtJ|!}: qtjtjC||tGtK|! rtjC||tGtK|!}:d}5ntjtjC||tDdV stjtjC||tE rtLdWtGtJ|!dX|dYntjtjC||tF rFtLdWtGtJ|!dX|dZnV| rjtLdWtGtH|!dX|d0n2tLdWtGtJ|!d[tEd[tDdVd\tFdX|d0 ntjtjC|| r|}:d}9ntjtjC||dV r| std]|dVd^tjC||dV}:d}9nvtM| r.|};tN|}<n\| r:tE};n,| rFtF};n | dk r\tGtH|!};n tGtJ|!};z|||||||,||d| d_ }=t||;f|=}<|tO|tEf|> rtL|dctGtJ|!dYntO|tFf|> rtL|dctGtJ|!dZnh|!dk rtO|tJf|> rtL|dctGtJ|!dd|!den,tL|dctGtJ|!d[tEd[tDd\tFd0 Wn`tLk r@YnJtPk rtLdf|dg|dhtGtJ|!d[tEd[tDd\tFd0 YnX|9 rt -di|:|:}(|/?|Djj_6d|D_i|0t%j8kr|/|D}D|'dk r|Dj|'|#||"d|r|7dkr|T|U|V|Xd}7|D|7fS|DS)a< Instantiate a pretrained pytorch model from a pre-trained model configuration. The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated). To train the model, you should first set it back in training mode with `model.train()`. The warning *Weights from XXX not initialized from pretrained model* means that the weights of XXX do not come pretrained with the rest of the model. It is up to you to train those weights with a downstream fine-tuning task. The warning *Weights from XXX not used in YYY* means that the layer XXX is not used by YYY, therefore those weights are discarded. Parameters: pretrained_model_name_or_path (`str` or `os.PathLike`, *optional*): Can be either: - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co. Valid model ids can be located at the root-level, like `bert-base-uncased`, or namespaced under a user or organization name, like `dbmdz/bert-base-german-cased`. - A path to a *directory* containing model weights saved using [`~PreTrainedModel.save_pretrained`], e.g., `./my_model_directory/`. - A path or url to a *tensorflow index checkpoint file* (e.g, `./tf_model/model.ckpt.index`). In this case, `from_tf` should be set to `True` and a configuration object should be provided as `config` argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards. - A path or url to a model folder containing a *flax checkpoint file* in *.msgpack* format (e.g, `./flax_model/` containing `flax_model.msgpack`). In this case, `from_flax` should be set to `True`. - `None` if you are both providing the configuration and state dictionary (resp. with keyword arguments `config` and `state_dict`). model_args (sequence of positional arguments, *optional*): All remaining positional arguments will be passed to the underlying model's `__init__` method. config (`Union[PretrainedConfig, str, os.PathLike]`, *optional*): Can be either: - an instance of a class derived from [`PretrainedConfig`], - a string or path valid as input to [`~PretrainedConfig.from_pretrained`]. Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when: - The model is a model provided by the library (loaded with the *model id* string of a pretrained model). - The model was saved using [`~PreTrainedModel.save_pretrained`] and is reloaded by supplying the save directory. - The model is loaded by supplying a local directory as `pretrained_model_name_or_path` and a configuration JSON file named *config.json* is found in the directory. state_dict (`Dict[str, torch.Tensor]`, *optional*): A state dictionary to use instead of a state dictionary loaded from saved weights file. This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using [`~PreTrainedModel.save_pretrained`] and [`~PreTrainedModel.from_pretrained`] is not a simpler option. cache_dir (`Union[str, os.PathLike]`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. from_tf (`bool`, *optional*, defaults to `False`): Load the model weights from a TensorFlow checkpoint save file (see docstring of `pretrained_model_name_or_path` argument). from_flax (`bool`, *optional*, defaults to `False`): Load the model weights from a Flax checkpoint save file (see docstring of `pretrained_model_name_or_path` argument). ignore_mismatched_sizes (`bool`, *optional*, defaults to `False`): Whether or not to raise an error if some of the weights from the checkpoint do not have the same size as the weights of the model (if for instance, you are instantiating a model with 10 labels from a checkpoint with 3 labels). force_download (`bool`, *optional*, defaults to `False`): Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist. resume_download (`bool`, *optional*, defaults to `False`): Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists. proxies (`Dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. output_loading_info(`bool`, *optional*, defaults to `False`): Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages. local_files_only(`bool`, *optional*, defaults to `False`): Whether or not to only look at local files (i.e., do not try to download the model). token (`str` or `bool`, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, or not specified, will use the token generated when running `huggingface-cli login` (stored in `~/.huggingface`). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. To test a pull request you made on the Hub, you can pass `revision="refs/pr/". mirror (`str`, *optional*): Mirror source to accelerate downloads in China. If you are from China and have an accessibility problem, you can set this option to resolve it. Note that we do not guarantee the timeliness or safety. Please refer to the mirror site for more information. _fast_init(`bool`, *optional*, defaults to `True`): Whether or not to disable fast initialization. One should only disable *_fast_init* to ensure backwards compatibility with `transformers.__version__ < 4.6.0` for seeded model initialization. This argument will be removed at the next major version. See [pull request 11471](https://github.com/huggingface/transformers/pull/11471) for more information. > Parameters for big model inference low_cpu_mem_usage(`bool`, *optional*): Tries to not use more than 1x model size in CPU memory (including peak memory) while loading the model. This is an experimental feature and a subject to change at any moment. torch_dtype (`str` or `torch.dtype`, *optional*): Override the default `torch.dtype` and load the model under a specific `dtype`. The different options are: 1. `torch.float16` or `torch.bfloat16` or `torch.float`: load in a specified `dtype`, ignoring the model's `config.torch_dtype` if one exists. If not specified - the model will get loaded in `torch.float` (fp32). 2. `"auto"` - A `torch_dtype` entry in the `config.json` file of the model will be attempted to be used. If this entry isn't found then next check the `dtype` of the first weight in the checkpoint that's of a floating point type and use that as `dtype`. This will load the model using the `dtype` it was saved in at the end of the training. It can't be used as an indicator of how the model was trained. Since it could be trained in one of half precision dtypes, but saved in fp32. For some models the `dtype` they were trained in is unknown - you may try to check the model's paper or reach out to the authors and ask them to add this information to the model's card and to insert the `torch_dtype` entry in `config.json` on the hub. device_map (`str` or `Dict[str, Union[int, str, torch.device]]` or `int` or `torch.device`, *optional*): A map that specifies where each submodule should go. It doesn't need to be refined to each parameter/buffer name, once a given module name is inside, every submodule of it will be sent to the same device. If we only pass the device (*e.g.*, `"cpu"`, `"cuda:1"`, `"mps"`, or a GPU ordinal rank like `1`) on which the model will be allocated, the device map will map the entire model to this device. Passing `device_map = 0` means put the whole model on GPU 0. To have Accelerate compute the most optimized `device_map` automatically, set `device_map="auto"`. For more information about each option see [designing a device map](https://hf.co/docs/accelerate/main/en/usage_guides/big_modeling#designing-a-device-map). max_memory (`Dict`, *optional*): A dictionary device identifier to maximum memory. Will default to the maximum memory available for each GPU and the available CPU RAM if unset. offload_folder (`str` or `os.PathLike`, *optional*): If the `device_map` contains any value `"disk"`, the folder where we will offload weights. offload_state_dict (`bool`, *optional*): If `True`, will temporarily offload the CPU state dict to the hard drive to avoid getting out of CPU RAM if the weight of the CPU state dict + the biggest shard of the checkpoint does not fit. Defaults to `True` when there is some disk offload. load_in_8bit (`bool`, *optional*, defaults to `False`): If `True`, will convert the loaded model into mixed-8bit quantized model. To use this feature please install `bitsandbytes` (`pip install -U bitsandbytes`). load_in_4bit (`bool`, *optional*, defaults to `False`): If `True`, will convert the loaded model into 4bit precision quantized model. To use this feature install the latest version of `bitsandbytes` (`pip install -U bitsandbytes`). quantization_config (`Union[QuantizationConfigMixin,Dict]`, *optional*): A dictionary of configuration parameters or a QuantizationConfigMixin object for quantization (e.g bitsandbytes, gptq) subfolder (`str`, *optional*, defaults to `""`): In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can specify the folder name here. variant (`str`, *optional*): If specified load weights from `variant` filename, *e.g.* pytorch_model..bin. `variant` is ignored when using `from_tf` or `from_flax`. use_safetensors (`bool`, *optional*, defaults to `None`): Whether or not to use `safetensors` checkpoints. Defaults to `None`. If not specified and `safetensors` is not installed, it will be set to `False`. kwargs (remaining dictionary of keyword arguments, *optional*): Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., `output_attentions=True`). Behaves differently depending on whether a `config` is provided or automatically loaded: - If a configuration is provided with `config`, `**kwargs` will be directly passed to the underlying model's `__init__` method (we assume all relevant updates to the configuration have already been done) - If a configuration is not provided, `kwargs` will be first passed to the configuration class initialization function ([`~PretrainedConfig.from_pretrained`]). Each key of `kwargs` that corresponds to a configuration attribute will be used to override said attribute with the supplied `kwargs` value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model's `__init__` function. Activate the special ["offline-mode"](https://huggingface.co/transformers/installation.html#offline-mode) to use this method in a firewalled environment. Examples: ```python >>> from transformers import BertConfig, BertModel >>> # Download model and configuration from huggingface.co and cache. >>> model = BertModel.from_pretrained("bert-base-uncased") >>> # Model was saved using *save_pretrained('./test/saved_model/')* (for example purposes, not runnable). >>> model = BertModel.from_pretrained("./test/saved_model/") >>> # Update configuration during loading. >>> model = BertModel.from_pretrained("bert-base-uncased", output_attentions=True) >>> assert model.config.output_attentions == True >>> # Loading from a TF checkpoint file instead of a PyTorch model (slower, for example purposes, not runnable). >>> config = BertConfig.from_json_file("./tf_model/my_tf_model_config.json") >>> model = BertModel.from_pretrained("./tf_model/my_tf_checkpoint.ckpt.index", from_tf=True, config=config) >>> # Loading from a Flax checkpoint file instead of a PyTorch model (slower) >>> model = BertModel.from_pretrained("bert-base-uncased", from_flax=True) ``` * `low_cpu_mem_usage` algorithm: This is an experimental function that loads the model using ~1x model size CPU memory Here is how it works: 1. save which state_dict keys we have 2. drop state_dict before the model is created, since the latter takes 1x model size CPU memory 3. after the model has been instantiated switch to the meta device all params/buffers that are going to be replaced from the loaded state_dict 4. load state_dict 2nd time 5. replace the params/buffers from the state_dict Currently, it can't handle deepspeed ZeRO stage 3 and ignores loading errors rNfrom_tfF from_flaxresume_downloadproxiesoutput_loading_infortrust_remote_codeZmirror_from_pipeline _from_auto _fast_initTrlow_cpu_mem_usager; max_memoryr<offload_state_dict load_in_8bit load_in_4bitquantization_config subfolderr _commit_hashrGadapter_kwargs adapter_namedefaultuse_flash_attention_2rrr rz0.37.2zgThe argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.) rIrKrRrSrLr rMr_%_raise_exceptions_for_missing_entriesZ'_raise_exceptions_for_connection_errors_adapter_model_path)rIrKrRrSrLr`rrrZbase_model_name_or_path)autoZbalancedbalanced_low_0 sequentialzWhen passing device_map as a string, the value needs to be a device name (e.g. cpu, cuda:0) or 'auto', 'balanced', 'balanced_low_0', 'sequential' but found rrznYou can't pass device_map as a negative int. If you want to put the model on the cpu, pass device_map = 'cpu' z>Passing along a `device_map` requires `low_cpu_mem_usage=True`z torch>=1.10z`DeepSpeed Zero-3 is not compatible with `low_cpu_mem_usage=True` or with passing a `device_map`.z^Using `low_cpu_mem_usage=True` or a `device_map` requires Accelerate: `pip install accelerate`Z quant_method)r\r])Z config_dictreturn_unused_kwargscSs&i|]\}}|ttjkr||qSrf)r7r8rHrrxrfrfrgr* sz3PreTrainedModel.from_pretrained..zYou can't pass `load_in_8bit` or any other `BitsAndBytesConfig` argument as a kwarg when passing `quantization_config` argument at the same time.zUsing `load_in_8bit=True` requires Accelerate: `pip install accelerate` and the latest version of bitsandbytes `pip install -i https://test.pypi.org/simple/ bitsandbytes` or pip install bitsandbytes` zOverriding torch_dtype=z with `torch_dtype=torch.float16` due to requirements of `bitsandbytes` to enable model loading in 8-bit or 4-bit. Pass your own torch_dtype to specify the dtype of the remaining non-linear layers or pass torch_dtype=torch.float16 to remove this warning.z/No GPU found. A GPU is needed for quantization.zThe device_map was not initialized.Setting device_map to {'':torch.cuda.current_device()}.If you want to use the model for inference, please set device_map ='auto' zConverting into 4-bit or 8-bit weights from tf/flax weights is currently not supported, please make sure the weights are in PyTorch format.rZpytorch) file_typerfrom_auto_classZusing_pipelinez+Offline mode: forcing local_files_only=True) rIrjrKrRrSrLr rMr_rWrVaSYou passed `quantization_config` to `from_pretrained` but the model you're loading already has a `quantization_config` attribute and has already quantized weights. However, loading attributes (e.g. disable_exllama, use_cuda_fp16, max_input_length) will be overwritten with the one you passed to `from_pretrained`. The rest will be ignored.z2GPU is required to quantize or run quantize model.zwLoading a GPTQ quantized model requires optimum (`pip install optimum`) and auto-gptq library (`pip install auto-gptq`)Z auto_gptqz0.4.2zWYou need a version of auto_gptq >= 0.4.2 to use GPTQ: `pip install --upgrade auto-gptq`) GPTQQuantizerzRWe suggest you to set `torch_dtype=torch.float16` for better efficiency with GPTQ.zYou passed `quantization_config` to `from_pretrained` but the model you're loading already has a `quantization_config` attribute. The `quantization_config` attribute will be overwritten with the one you passed to `from_pretrained`.)rjz(Invalid type for `quantization_config`: z8. Should be a `dict` or a `BitsAndBytesConfig` instance.a Detected the presence of a `quantization_config` attribute in the model's configuration but you don't have the correct `bitsandbytes` version to support int8 serialization. Please install the latest version of `bitsandbytes` with `pip install --upgrade bitsandbytes`.z.indexzError no file named z found in directory zf but there is a file for TensorFlow weights. Use `from_tf=True` to load this model from those weights.zb but there is a file for Flax weights. Use `from_flax=True` to load this model from those weights.z, rz$We found a TensorFlow checkpoint at z:, please set from_tf to True to load from this checkpoint.) rIrKrSrRrLr  user_agentrMr_rer` z and thus cannot be loaded with `safetensors`. Please make sure that the model has been saved with `safe_serialization=True` or do not set `use_safetensors=True`.)rMrSr z& does not appear to have a file named z) but there is a file without the variant z;. Use `variant=None` to load this model from those weights.zCan't load the model for 'z'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'z=' is the correct path to a directory containing a file named zloading weights file z from cache at ) rIrKrSrRrLr rnrMr_r`rgzWill use torch_dtype=z$ as defined in model's config objectrzSince the `torch_dtype` attribute can't be found in model's config object, will use torch_dtype={torch_dtype} as derived from model's weightszD`torch_dtype` can be either `torch.dtype` or `"auto"`, but received Zall_checkpoint_keys)rprr)rr;r)get_keys_to_not_convertreplace_with_bnb_linearz?Detected 8-bit loading: activating 8-bit loading for this modelz?Detected 4-bit loading: activating 4-bit loading for this modelcSsg|]\}}|dkr|qS))r1rrfryrr0rfrfrgr|. sz3PreTrainedModel.from_pretrained..zIf you want to offload some keys to `cpu` or `disk`, you need to set `llm_int8_enable_fp32_cpu_offload=True`. Note that these modules will not be converted to 8-bit but kept in 32-bit.z0.39.0zYou have a version of `bitsandbytes` that is not compatible with 4bit inference and training make sure you have the latest version of `bitsandbytes` installed)modules_to_not_convertr^z0.37.0zYou are loading your model in 8bit but you did not specify a `torch_dtype` attribute.All non-linear modules will be loaded in full precision. If you want to load the other modules in other precision, please specify a `torch_dtype` attribute.rCcs,i|]$\}tfddDrqS)c3s|]}|kVqdSr,rfrrrfrgr/d s=PreTrainedModel.from_pretrained...)r5ryr:)rsrrrgra scs.i|]&\}tfddDrtjqS)c3s|]}|kVqdSr,rfrrrfrgr/l srt)r5r_rru)rBrrgri s acceleratez0.19.0) CustomDtypeaUYou are using `device_map='auto'` on a 4bit loaded version of the model. To automatically compute the appropriate device map, you should upgrade your `accelerate` library,`pip install --upgrade accelerate` or install it from source to support fp4 auto device mapcalculation. You may encounter unexpected behavior, or pass your own device mapz does not support `device_map='z^'`. To implement support, the modelclass needs to implement the `_no_split_modules` attribute.ziIf passing a string for `device_map`, please choose 'auto', 'balanced', 'balanced_low_0' or 'sequential'.Zno_split_module_classesspecial_dtypeszThis model has some weights that should be kept in higher precision, you need to upgrade `accelerate` to properly deal with them (`pip install --upgrade accelerate`).rirh)rZlow_zerorZcSsi|]\}}||dqS)g?rf)ryrvalrfrfrgr scsi|]}|kr||qSrfrfr)r;rsrfrgr srr1ad Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the quantized model. If you want to dispatch the model on the CPU or the disk while keeping these modules in 32-bit, you need to set `load_in_8bit_fp32_cpu_offload=True` and pass a custom `device_map` to `from_pretrained`. Check https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu for more details. i)$load_tf2_checkpoint_in_pytorch_model)Zallow_missing_keysrTzLoading a TensorFlow model in PyTorch, requires both PyTorch and TensorFlow to be installed. Please see https://pytorch.org/ and https://www.tensorflow.org/install/ for installation instructions.)%load_flax_checkpoint_in_pytorch_modelzLoading a Flax model in PyTorch, requires both PyTorch and Flax to be installed. Please see https://pytorch.org/ and https://flax.readthedocs.io/en/latest/installation.html for installation instructions.) rJsharded_metadatarXrYr;r<r[rr@rB) rIrKrRrSrLr rMr_rWrVzZGeneration config file not found, using a generation config created from the model config.)r;Z offload_dirr= skip_keysrz%We can only quantize pure text model.)rbr ra)rrmismatched_keysr)rrhrjrkrlrr=r7r rrrrrrrr0r$r3rr;rnrcrrrrrr_rrrrjrKrr5rKrJrDrH from_dictr\r]rrrr6rraZcurrent_devicer9 config_classfrom_pretrainedr r^reZGPTQZget_loading_attributesr:r6Z optimum.gptqrmrIto_dictr risdirrr*r)r&rHr(r'r,r+EnvironmentErrorr<r2r4rrCrrrr_keep_in_fp32_modulesrrrrqr r rrrrQrr-r4rprqllm_int8_skip_modulesZ llm_int8_enable_fp32_cpu_offloadextendZ_is_quantized_training_enabledrurZ convert_modelrCr@updateraccelerate.utilsrwZINT4r9_no_split_modulesrrr7r8rPrrUrVrrrTrSrZload_tf_weightsZmodeling_tf_pytorch_utilsrzr#Zmodeling_flax_pytorch_utilsr{r_load_pretrained_modelrrevalrrrrrO_skip_keys_device_placementZ tokenizerrZquantize_modelZpost_init_modelZ load_adapter)YrrOrurIrJrKrLr rMrNZ model_argsrQrrPrQrRrSrTrrUr:Z from_pipelinerlrXrYrZr<r[r\r]r^r_Z commit_hashrGrarbrdrZresolved_config_filerfrZquantization_method_from_argsZquantization_config_kwargsZfrom_ptrnZ config_pathZ model_kwargsZ quantizerZquantization_method_from_configZloading_attr_dictattrryrmZ is_shardedr|Z loading_infoZuse_keep_in_fp32_modulesis_local archive_filer=resolved_archive_fileZcached_file_kwargsZhas_file_kwargsrZone_state_dictr&Z init_contextsr rrprqrZload_in_8bit_fp32_cpu_offloadZ keys_on_cpuZ supports_4bitrx target_dtyperwZno_split_modulesZdevice_map_kwargsZdevice_map_without_lm_head tied_paramsrzr{rrr~r=rrf)r;rBrsrrgrsv                                                                                 ,           &.                                                                zPreTrainedModel.from_pretrainedc5s d}|rddlm}| dk rd| krt|ttfr>|dn|d}| dkr`|s`td| dk rvtj | dd | dkrd} |o|dk }| | }t| }|j d d |}fd d |D}t dkr t fdd|D}t fdd|D}nd}d}| o|}|o(| }|r` dfdd |Dfdd |D}n|rx fdd |D}tt|t|}t|t|}dd|D}|rȇfdd|D}n|r fdd|D}t||}| | dkrRtsRtt}| D] \ }t|}|| qdd |D} nt|} | D]||r|fdd Dn|r fdd Dfdd |D t dkr^t tkr^ fdd |D}q^jdk rjD] fdd |D}qjdk r4jD] fdd |D}q| rl|D]*t| kr\nv dt| kr dnJ rddddt| krdddd|}!| }"|dk r| tjkrtfd d|Drtj }"|!j!t!d!kr>|sJt"|d"tj#|!$d#|"in||d"tj#|!$d#|"iq>|r|r fd$d |D}#n|r fd%d |D}#n|}#t%||#|&|j'|dk r|(D]2\ }!t fd&d|Dr|!j)*tj |!_)qd'}$|}%tj dkrtj-| |0d:tj-| d|0d:q fd;d+|(D}(| st?|(| d}(| r4t@|%|,|+t=A|+t|*dk rvd<|*}1d=|1k r^|1d>7}1tBd?|jCjDd@|1| rdAd |D}dBd |D}t|dk r|jEjFdk rgn|jEjF}2|jCjD|2k rtGjHntGjI}3|3dC|dD|jCjDdE|dF|jCjDdG|jCjDdH ntGIdI|jCjDdJt|dk rXtGHdK|jCjDdL|dM|dNn4t|)dk rtGIdO|jCjDdP|dQ|jCjDdRt|)dk rdSdTd |)D}4tGHdK|jCjDdL|dU|4dN||||)|(|*fS)VNFrr*r1rrzThe current `device_map` had weights offloaded to the disk. Please provide an `offload_folder` for them. Alternatively, make sure you have `safetensors` installed if the model you are using offers the weights in this format.TrcSs,d|kr|ddSd|kr(|ddS|S)Nrrrr)rrrfrfrg_fix_keyx s   z8PreTrainedModel._load_pretrained_model.._fix_keycsg|] }|qSrfrfr)rrfrgr| sz:PreTrainedModel._load_pretrained_model..c3s|]}|VqdSr,rrysrrfrgr/ sz9PreTrainedModel._load_pretrained_model..c3s|]}|VqdSr,rrrrfrgr/ srcsg|]}|s|qSrfrr_prefixrfrgr| s cs*g|]"}|r"|tdn|qSr,rrrrrfrgr| scsg|]}d|gqSrrrrrfrgr| scSsh|] \}}|qSrfrf)ryrr:rfrfrgr sz9PreTrainedModel._load_pretrained_model..cs*h|]"}|r"|tdn|qSr,rrrrfrgr scsh|]}d|gqSrrrrrfrgr scSs g|]\}}t|dkr|qSrr)ryr:rrfrfrgr| s cs*g|]"}|r"|tdn|qSr,rrrrfrgr| scsg|]}d|gqSrrrrrfrgr| scsg|]}|kr|qSrfrfr)grouprfrgr| scsg|]}|kr|qSrfrfr)missing_in_grouprfrgr| scs g|]}t|dkr|qSr,rrrrfrgr| scs g|]}t|dkr|qSr,rrrrfrgr| sc3s|]}|kVqdSr,rfr-rrfrgr/ srrrcsg|]}d|qSrrfrrrfrgr| scs g|]}|tddqS)rNrrrrfrgr| sc3s|]}|kVqdSr,rfr-rrfrgr/ src3s|]}|ko|kVqdSr,rfr)base_model_expected_keysexpected_keys_not_prefixedrfrgr/ szjThe state dictionary of the model you are trying to load is corrupted. Are you sure it was properly saved?cs&i|]\}}|jdd|qSr)rrrxrrfrgr sz:PreTrainedModel._load_pretrained_model..c sg}|r|D]}||krq |}|r2d|}n|rNd|ddd}||kr ||j||jkr ||||j||jf||=q |S)Nrr)rrrnr) rmodel_state_dictradd_prefix_to_modelremove_prefix_from_modelrJr~Zcheckpoint_keyZ model_keyrrfrg_find_mismatched_keyss&zEPreTrainedModel._load_pretrained_model.._find_mismatched_keysriztorch.rcsi|] }|qSrfrfryp)rrfrgr.scs i|]\}}|tj|qSrfrcrrryrrrrfrgr0srcs*i|]"\}}|dkr|||dqS)r1)Zsafetensors_file weight_namerrfr)param_device_map str_dtyperfrgr1s )r|csg|]}tj|qSrfr)ryrrrfrgr|WszLoading checkpoint shards)desc) r;r<r=r>r?rr@rArBz.datcs i|]\}}d||qSrrfrrrrfrgrs z z size mismatchz_ You may consider adding `ignore_mismatched_sizes=True` in the model `from_pretrained` method.rz: cSsg|]}d|kr|qSr2rfryelemrfrfrgr|scSsg|]}d|kr|qSrrfrrfrfrgr|sz(Some weights of the model checkpoint at z! were not used when initializing z: z, - This IS expected if you are initializing z from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing z from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).z9All model checkpoint weights were used when initializing z. zSome weights of z3 were not initialized from the model checkpoint at z and are newly initialized: zo You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.zAll the weights of z/ were initialized from the model checkpoint at zf. If your task is similar to the task the model of the checkpoint was trained on, you can already use z* for predictions without further training.r"c Ss*g|]"\}}}d|d|d|dqS)z- z: found shape z in the checkpoint and z in the model instantiatedrf)ryrZshape1Zshape2rfrfrgr|sz= and are newly initialized because the shapes did not match: )Jr4r+rrrrrrrcr$rrrrrr5rZ named_buffersrhr.r/rrrrT_keys_to_ignore_on_load_missing"_keys_to_ignore_on_load_unexpectedrrrr_r6rrrZemptyrrrrrrr$r rrr%expand_device_maprrrtempfilemkdtempget_disk_only_shard_filesr?ZtqdmrrkrFrrshutilmoverYrWrmtreerrrrur'rrr)5rrrrrrOrJr|rXrYr;r<r[rr@rBrAr+Zis_sharded_safetensorsrr:Zoriginal_loaded_keysZhas_prefix_moduleZexpects_prefix_modulerrrrZ model_buffersr;rZ id_tensorrrCrZ _loaded_keysrrrrr=r~rr>r?Zdisk_only_shard_filesrZnew_error_msgsr error_msgZarchsZwarnerZmismatched_warningrf)rrrrrrrrrrrrrrrrgrG s                 2    $ $   &                     . z&PreTrainedModel._load_pretrained_modelc Csdd|D}|dd|D}g}|D]p\}}|rf|jd}||r`|t|dn|}n&|rt|dkrd|j|gn|j}||kr.||q.|S)NcSs$h|]}d|dddqS)rNri)rrrrfrfrgrsz>PreTrainedModel.retrieve_modules_from_names..cSs<h|]4}t|dkr|drd|dddqS)rrirN)rrrrrrfrfrgrs rr)unionrrrrrr) r[rZ add_prefixZ remove_prefixZ module_keysZretrieved_modulesrrurrfrfrgretrieve_modules_from_namess   " z+PreTrainedModel.retrieve_modules_from_namescCs&t|||t|}t||||}|S)a~ This is an experimental function that loads the model using ~1.x model size CPU memory Before you call it do: 1. save which state_dict keys are available 2. drop state_dict before model is created, since the latter takes 1x model size memory Here then we continue: 3. switch to the meta device all params/buffers that are going to be replaced from the loaded state_dict 4. load state_dict 2nd time 5. replace the params/buffers from the state_dict Currently, it doesn't handle missing_keys, unexpected_keys, mismatched_keys. It can't handle deepspeed. )r)rrF)rr&rrrrrfrfrg_load_pretrained_model_low_mems z.PreTrainedModel._load_pretrained_model_low_mem AutoModelcCsDt|ts|j}ddlmm}t||s:t|d||_dS)a Register this class with a given auto class. This should only be used for custom models as the ones in the library are already mapped with an auto class. This API is experimental and may have some slight breaking changes in the next releases. Args: auto_class (`str` or `type`, *optional*, defaults to `"AutoModel"`): The auto class to register this new model with. rNz is not a valid auto class.) rrrZtransformers.models.automodelsrgr rr()rZ auto_classZ auto_modulerfrfrgregister_for_auto_class s   z'PreTrainedModel.register_for_auto_classcCsTtstdddlm}t|tdkr>td|dddlm}||S)a( Converts the model to use [PyTorch's native attention implementation](https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html), integrated to Transformers through [Optimum library](https://huggingface.co/docs/optimum/bettertransformer/overview). Only a subset of all Transformers models are supported. PyTorch's attention fastpath allows to speed up inference through kernel fusions and the use of [nested tensors](https://pytorch.org/docs/stable/nested.html). Detailed benchmarks can be found in [this blog post](https://medium.com/pytorch/bettertransformer-out-of-the-box-performance-for-huggingface-transformers-3fbe27d50ab2). Returns: [`PreTrainedModel`]: The model converted to BetterTransformer. =1.7.0 to use Better Transformer. The version was found.BetterTransformer) r:rKoptimum.versionrmr roptimum.bettertransformerrZ transformr[Zoptimum_versionrrfrfrgto_bettertransformer&s   z$PreTrainedModel.to_bettertransformercCsTtstdddlm}t|tdkr>td|dddlm}||S)a Reverts the transformation from [`~PreTrainedModel.to_bettertransformer`] so that the original modeling is used, for example in order to save the model. Returns: [`PreTrainedModel`]: The model converted back to the original modeling. rrrlrrrr) r:rKrrmr rrrreverserrfrfrgreverse_bettertransformerBs   z)PreTrainedModel.reverse_bettertransformerc Cst|stjstrdS|dk s0|jjdkr4dS|jj|ddddgfkrd}|jjdk rp|jj|jjks|jjdk r|jj|jjks|jj dk r|jj |jjkr|d|jjd|jjd|jjd|jj d 7}t |dS) zv Shows a one-time warning if the input_ids appear to contain padding and no attention mask was given. NrirzWe strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked.z5 You may ignore this warning if your `pad_token_id` (z&) is identical to the `bos_token_id` (z), `eos_token_id` (z), or the `sep_token_id` (z ), and your input is not padded.) rFr_Zjit is_tracingrGruZ pad_token_idZ bos_token_idZ eos_token_idZ sep_token_idrr3)r[rrqZ warn_stringrfrfrg%warn_if_padding_and_no_attention_maskXs.   ,z5PreTrainedModel.warn_if_padding_and_no_attention_mask)NN)NN)N)NN)NF)T) FNTFNNNNFN)FF)r)r)Zrrrrrrrr(rrrrrr-r0Zis_parallelizablerrrrrr_rrrrrrr classmethodrrrrrrrrr r rjrrrrrrrorrrrrrrrrrrrrr rrr rrrrsavercPathLiker rr)rBrrr$rFrrrrrrrrr __classcell__rfrfrrgr-sH  M     J &  h S          6    rrrz model file)objectZ object_classZ object_filescsDeZdZdZedfdd Zd ejeejejdddZ Z S) PoolerStartLogitsz Compute SQuAD start logits from sequence hidden states. Args: config ([`PretrainedConfig`]): The config used by the model, will be used to grab the `hidden_size` of the model. rcstt|jd|_dS)Nr)rrrr hidden_sizedenser[rurrfrgrs zPoolerStartLogits.__init__N) hidden_statesp_maskrvcCsT||d}|dk rPt|tjkr<|d|d|}n|d|d|}|S)a Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, seq_len, hidden_size)`): The final hidden states of the model. p_mask (`torch.FloatTensor` of shape `(batch_size, seq_len)`, *optional*): Mask for tokens at invalid position, such as query and special symbols (PAD, SEP, CLS). 1.0 means token should be masked. Returns: `torch.FloatTensor`: The start logits for SQuAD. riNrꌠ9Y>)F)rsqueezerr_r6)r[rrxrfrfrgforwards zPoolerStartLogits.forward)N) rrrrrrr_ FloatTensorr rrrfrfrrgrsrcsTeZdZdZedfdd Zd ejeejeej eejejdddZ Z S) PoolerEndLogitsz Compute SQuAD end logits from sequence hidden states. Args: config ([`PretrainedConfig`]): The config used by the model, will be used to grab the `hidden_size` of the model and the `layer_norm_eps` to use. rcsRtt|jd|j|_t|_tj|j|j d|_t|jd|_ dS)Nra)Zepsr) rrrrrdense_0Tanh activation LayerNormZlayer_norm_epsdense_1rrrfrgrs   zPoolerEndLogits.__init__N)r start_statesstart_positionsrrvcCs|dk s|dk std|dk rh|jdd\}}|ddddfdd|}|d|}|d|d}|tj||gdd}||}||}| | d}|dk rt |tj kr|d|d|}n|d|d|}|S) a Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, seq_len, hidden_size)`): The final hidden states of the model. start_states (`torch.FloatTensor` of shape `(batch_size, seq_len, hidden_size)`, *optional*): The hidden states of the first tokens for the labeled span. start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): The position of the first token for the labeled span. p_mask (`torch.FloatTensor` of shape `(batch_size, seq_len)`, *optional*): Mask for tokens at invalid position, such as query and special symbols (PAD, SEP, CLS). 1.0 means token should be masked. One of `start_states` or `start_positions` should be not `None`. If both are set, `start_positions` overrides `start_states`. Returns: `torch.FloatTensor`: The end logits for SQuAD. N7One of start_states, start_positions should be not Nonerrirdrrr) rrnr~gatherrr_rorrrrrr6)r[rrrrslenhszrrfrfrgrs(   zPoolerEndLogits.forward)NNN rrrrrrr_rr LongTensorrrrfrfrrgrs  rcsNeZdZdZfddZdejeejeejeejejdddZ Z S) PoolerAnswerClassz Compute SQuAD 2.0 answer class from classification and start tokens hidden states. Args: config ([`PretrainedConfig`]): The config used by the model, will be used to grab the `hidden_size` of the model. csBtt|jd|j|_t|_tj|jddd|_dS)NrarF)r) rrrrrrrrrrrrfrgrs  zPoolerAnswerClass.__init__N)rrr cls_indexrvcCs|jd}|dk s"|dk s"td|dk rX|ddddfdd|}|d|d}|dk r|ddddfdd|}|d|d}n|dddddf}|tj||gdd}||}| |d}|S)a Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, seq_len, hidden_size)`): The final hidden states of the model. start_states (`torch.FloatTensor` of shape `(batch_size, seq_len, hidden_size)`, *optional*): The hidden states of the first tokens for the labeled span. start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): The position of the first token for the labeled span. cls_index (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Position of the CLS token for each sentence in the batch. If `None`, takes the last token. One of `start_states` or `start_positions` should be not `None`. If both are set, `start_positions` overrides `start_states`. Returns: `torch.FloatTensor`: The SQuAD 2.0 answer class. riNrrr) rnrr~rrrr_rorr)r[rrrrrZcls_token_staterrfrfrgrs$  zPoolerAnswerClass.forward)NNN) rrrrrr_rr rrrrfrfrrgrs rc@s~eZdZUdZdZeejed<dZ eejed<dZ eej ed<dZ eejed<dZ eej ed<dZeejed<dS) SquadHeadOutputa Base class for outputs of question answering models using a [`~modeling_utils.SQuADHead`]. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned if both `start_positions` and `end_positions` are provided): Classification loss as the sum of start token, end token (and is_impossible if provided) classification losses. start_top_log_probs (`torch.FloatTensor` of shape `(batch_size, config.start_n_top)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Log probabilities for the top config.start_n_top start token possibilities (beam-search). start_top_index (`torch.LongTensor` of shape `(batch_size, config.start_n_top)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Indices for the top config.start_n_top start token possibilities (beam-search). end_top_log_probs (`torch.FloatTensor` of shape `(batch_size, config.start_n_top * config.end_n_top)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Log probabilities for the top `config.start_n_top * config.end_n_top` end token possibilities (beam-search). end_top_index (`torch.LongTensor` of shape `(batch_size, config.start_n_top * config.end_n_top)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Indices for the top `config.start_n_top * config.end_n_top` end token possibilities (beam-search). cls_logits (`torch.FloatTensor` of shape `(batch_size,)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Log probabilities for the `is_impossible` label of the answers. Nlossstart_top_log_probsstart_top_indexend_top_log_probs end_top_index cls_logits)rrrrrr r_r__annotations__rrrrrrrfrfrfrgr/s rc sxeZdZdZfddZeeedd ej e ej e ej e ej e ej e ej e e eeej fddd ZZS) SQuADHeadz A SQuAD head inspired by XLNet. Args: config ([`PretrainedConfig`]): The config used by the model, will be used to grab the `hidden_size` of the model and the `layer_norm_eps` to use. cs<t|j|_|j|_t||_t||_t||_ dSr,) rr start_n_top end_n_topr start_logitsr end_logitsr answer_classrrrfrgrXs    zSQuADHead.__init__) output_typerNF)rr end_positionsr is_impossibler return_dictrvcCs|j||d}|dk r|dk r||||fD]"} | dk r*| dkr*| dq*|j|||d} t} | ||} | | |} | | d}|dk r|dk r|j|||d}t}|||}||d7}|rt|d S|fS| \}}}tj j |dd }t j ||jdd \}}|ddd|}t |d |}|dd|dd}|d|}|dk rf|dnd}|j|||d } tj j | dd }t j ||jdd \}}|d|j|j}|d|j|j}t d ||}|j|||d}|s|||||fSt|||||dSdS)a Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, seq_len, hidden_size)`): Final hidden states of the model on the sequence tokens. start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Positions of the first token for the labeled span. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Positions of the last token for the labeled span. cls_index (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Position of the CLS token for each sentence in the batch. If `None`, takes the last token. is_impossible (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Whether the question has a possible answer in the paragraph or not. p_mask (`torch.FloatTensor` of shape `(batch_size, seq_len)`, *optional*): Mask for tokens at invalid position, such as query and special symbols (PAD, SEP, CLS). 1.0 means token should be masked. return_dict (`bool`, *optional*, defaults to `False`): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Returns: )rNrri)rrra)rrg?)rrr)rrz blh,bl->bh)rr)rrrrr)rrdZsqueeze_rrrrZBCEWithLogitsLossrrrZsoftmaxr_Ztopkrr{r~rZ expand_asrviewZeinsum)r[rrrrrrrrrrZloss_fctZ start_lossZend_lossZ total_lossrZ loss_fct_clsZcls_lossZbszrrZstart_log_probsrrZstart_top_index_exprZhidden_states_expandedZ end_log_probsrrrfrfrgrasd         zSQuADHead.forward)NNNNNF)rrrrrr@rrr_rr rrr r rrrfrfrrgrNs& rcsDeZdZdZedfdd Zd ejeej ejdddZ Z S) SequenceSummarya Compute a single vector summary of a sequence hidden states. Args: config ([`PretrainedConfig`]): The config used by the model. Relevant arguments in the config class of the model are (refer to the actual config class of your model for the default values it uses): - **summary_type** (`str`) -- The method to use to make this summary. Accepted values are: - `"last"` -- Take the last token hidden state (like XLNet) - `"first"` -- Take the first token hidden state (like Bert) - `"mean"` -- Take the mean of all tokens hidden states - `"cls_index"` -- Supply a Tensor of classification token position (GPT/GPT-2) - `"attn"` -- Not implemented now, use multi-head attention - **summary_use_proj** (`bool`) -- Add a projection after the vector extraction. - **summary_proj_to_labels** (`bool`) -- If `True`, the projection outputs to `config.num_labels` classes (otherwise to `config.hidden_size`). - **summary_activation** (`Optional[str]`) -- Set to `"tanh"` to add a tanh activation to the output, another string or `None` will add no activation. - **summary_first_dropout** (`float`) -- Optional dropout probability before the projection and activation. - **summary_last_dropout** (`float`)-- Optional dropout probability after the projection and activation. rcstt|dd|_|jdkr&tt|_t|drv|jrvt|dr`|j r`|j dkr`|j }n|j }t |j ||_t|dd}|rt|nt|_t|_t|dr|jdkrt |j|_t|_t|d r|jdkrt |j|_dS) N summary_typelastattnsummary_use_projsummary_proj_to_labelsrZsummary_activationsummary_first_dropoutsummary_last_dropout)rrrrrrsummaryr r r Z num_labelsrrrrr first_dropoutr ZDropout last_dropoutr)r[ruZ num_classesZactivation_stringrrfrgrs$   zSequenceSummary.__init__N)rrrvcCs|jdkr|dddf}n|jdkr8|dddf}n|jdkrP|jdd}n|jd kr|dkrtj|d ddddf|jd dtjd }n2|dd}|d |d| df}| d | d }n|jdkrt | |}||}||}||}|S)ak Compute a single vector summary of a sequence hidden states. Args: hidden_states (`torch.FloatTensor` of shape `[batch_size, seq_len, hidden_size]`): The hidden states of the last layer. cls_index (`torch.LongTensor` of shape `[batch_size]` or `[batch_size, ...]` where ... are optional leading dimensions of `hidden_states`, *optional*): Used if `summary_type == "cls_index"` and takes the last token of the sequence as classification token. Returns: `torch.FloatTensor`: The summary of the sequence hidden states. r Nrifirstrmeanrrr.rrb)rir )rrr_Z full_likernlongr{r~rdrrrrrrrr)r[rrrrfrfrgrs.     "     zSequenceSummary.forward)Nrrfrfrrgrsr)rrvcCst|drt|jS|SdS)z Recursively unwraps a model from potential containers (as used in distributed training). Args: model (`torch.nn.Module`): The model to unwrap. ruN)r r&ru)rrfrfrgr&&s  r&cs4i}|D]"\|fdd|Dq |S)zT Expand a device map to return the correspondance parameter name to device. cs*i|]"}|ks |dr|qSrrrrrurfrgr:s z%expand_device_map..)rr)r;Z param_namesZnew_device_maprfrrgr4srcCsrtt}|dD]H\}}t|dkrL||krLd|ddd}q||||qdd|DS)zT Returns the list of shard files containing only weights offloaded to disk. rrrNricSs"g|]\}}t|dhkr|qS)r1)r)ryfnameZdevicesrfrfrgr|Hsz-get_disk_only_shard_files..)r.r/rrrrrr)r;r|Z files_contentrr=rfrfrgr>s  r)T)TT) NNNNNNFFN)N)r.rimportlib.metadatarr7rrcrrrrj contextlibr dataclassesr functoolsrrtypingrrrr r r r r_ packagingr rrZtorch.nnrrZ activationsrZconfiguration_utilsrZdynamic_module_utilsrZ generationrrr4rrrZ pytorch_utilsrrrrrr r!utilsr"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=r>r?r@rAZ utils.hubrBrCZutils.import_utilsrDrErFrGZutils.quantization_configrHrIrJZutils.versionsrKrdreupperrLrNrvrOrPrQZaccelerate.hooksrRrrSrTrUrVrWrXrYrZZ safetensorsr[Zsafetensors.torchr\rr]r7Z get_loggerrrrorhrkZ!smdistributed.modelparallel.torchZ modelparallelr+Zsmdistributed.modelparallelrmZ SMP_VERSIONrr*rnrqrrrrrrrrrjrrrrrrr"r)rFrHrrrr rrrrrrrrr&rrrfrfrfrgs  $    $ "  (        1    S T2 =   2e  &EAuc