U 9%e5@sddlZddlZddlmZddlmZddlmZm Z m Z e rjddl Z ddl Z ddlmZddlmZerddlmZddlmZe eZdd d Zdd d ZdddZddZddZddZdS)N)deepcopy)version)is_accelerate_availableis_bitsandbytes_availablelogging)Conv1D)init_empty_weights)find_tied_parameterscCsd|krV|d}|ddD].}t||}|dkrHt|d|d|}q|d}||jkr~||jkr~t|d|d||jk}t||} | jtdkr|dtdfkr|dkrt|d|dd} d} |stsd} d} n6tt j d o t |j|t j j } t |j|t j j } | s.| rT|j|} | jjd kr|dkr\| |} n`t |tjr|d } |jtjkrttjd td k}|stdntj|d d} t|jtr|dkr| j} | j}| rt j j | fddi||} n$| r*t j j | fddi||} | |j|<|dk rt|jd||nj|dkrj| |} n(t |tjr||} ntj||d} |r| |j|<nt j| | j d} | |j|<dS)a A helper function to set a given tensor (parameter of buffer) of a module on a specific device (note that doing `param.to(device)` creates a new tensor not linked to the parameter, which is why we need this function). The function is adapted from `set_module_tensor_to_device` function from accelerate that is adapted to support the class `Int8Params` from `bitsandbytes`. Args: module (`torch.nn.Module`): The module in which the tensor we want to move lives. tensor_name (`str`): The full name of the parameter/buffer. device (`int`, `str` or `torch.device`): The device on which to set the tensor. value (`torch.Tensor`, *optional*): The value of the tensor (useful when going from the meta device to any other device). fp16_statistics (`torch.HalfTensor`, *optional*): The list of fp16 statistics to set on the module, used for serialization. .Nz has no attribute z- does not have a parameter or a buffer named metaz7 is on the meta device, we need a `value` to put in on F Params4bitcudacpu bitsandbytesz0.37.2zDetected int8 weights but the version of bitsandbytes is not compatible with int8 serialization. Make sure to download the latest `bitsandbytes` version. `pip install --upgrade bitsandbytes`.)device requires_gradZSCB)r)!splitgetattr ValueError _parameters_buffersrtorchrhasattrbnbnn isinstancerZ Int8ParamstypetoZTensorZdtypeZint8rparse importlibmetadataZtensor issubclass source_clsrT__dict__setattrweight Parameterr)moduleZ tensor_namervalueZfp16_statisticsZsplitsrZ new_moduleZ is_buffer old_valueZis_4bitZis_8bitparam new_valueZis_8bit_serializablekwargsr0e/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/transformers/integrations/bitsandbytes.py%set_module_quantized_tensor_to_devicesp    * "            r2Fc st|D]`\}}dkrg|t|tjs@t|tr4||kr4tfdd|Ds4tt|tr|jj \}}n |j }|j }| dkrt jj|||jdk |j|jd|j|<d}nD|jdk r||jkrn.t jj|||jdk |j|j|jd|j|<d}t||j|_|j|dW5QRXtt|d kr`t||||d \} }d q||fS) z Private method that wraps the recursion for module replacement. Returns the converted model and a boolean that indicates if the conversion has been successfull or not. Nc3s|]}|dkVqdS)r N)join).0keycurrent_key_namer0r1 sz+_replace_with_bnb_linear..Zllm_int8)Zhas_fp16_weights thresholdT)Zcompress_statisticsZ quant_typeFr)has_been_replacedr ) Znamed_childrenappendrrZLinearranyr r(shape in_features out_featuresZquantization_methodrZ Linear8bitLtZbiasZllm_int8_has_fp16_weightZllm_int8_thresholdZ_modulesZllm_int8_skip_modulesZ Linear4bitZbnb_4bit_compute_dtypeZbnb_4bit_use_double_quantZbnb_4bit_quant_typerr$Zrequires_grad_lenlistchildren_replace_with_bnb_linearpop) modelmodules_to_not_convertr7quantization_configr:namer*r>r?_r0r6r1rCws\ "      rCcCs6|dkrdgn|}t||||\}}|s2td|S)a6 A helper function to replace all `torch.nn.Linear` modules by `bnb.nn.Linear8bit` modules from the `bitsandbytes` library. This will enable running your models using mixed int8 precision as described by the paper `LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale`. Make sure `bitsandbytes` compiled with the correct CUDA version of your hardware is installed before running this function. `pip install -i https://test.pypi.org/simple/ bitsandbytes` The function will be run recursively and replace all `torch.nn.Linear` modules except for the `lm_head` that should be kept as a `torch.nn.Linear` module. The replacement is done under `init_empty_weights` context manager so no CPU/GPU memory is required to run this function. Int8 mixed-precision matrix decomposition works by separating a matrix multiplication into two streams: (1) and systematic feature outlier stream matrix multiplied in fp16 (0.01%), (2) a regular stream of int8 matrix multiplication (99.9%). With this method, int8 inference with no predictive degradation is possible for very large models (>=176B parameters). Parameters: model (`torch.nn.Module`): Input model or `torch.nn.Module` as the function is run recursively. modules_to_not_convert (`List[`str`]`, *optional*, defaults to `["lm_head"]`): Names of the modules to not convert in `Linear8bitLt`. In practice we keep the `lm_head` in full precision for numerical stability reasons. current_key_name (`List[`str`]`, *optional*): An array to track the current key of the recursion. This is used to check whether the current key (part of it) is not in the list of modules to not convert (for instances modules that are offloaded to `cpu` or `disk`). NZlm_headzYou are loading your model in 8bit or 4bit but no linear modules were found in your model. Please double check your model architecture, or submit an issue on github if you think this is a bug.)rCloggerwarning)rErFr7rGr:r0r0r1replace_with_bnb_linearsrLcOstdtt||S)Nzj`replace_8bit_linear` will be deprecated in a future version, please use `replace_with_bnb_linear` instead)warningswarn FutureWarningrLargsr/r0r0r1replace_8bit_linears rRcOstdtt||S)Nz`set_module_8bit_tensor_to_device` will be deprecated in a future version, please use `set_module_quantized_tensor_to_device` instead)rMrNrOr2rPr0r0r1 set_module_8bit_tensor_to_devices rSc st|}|t|}t|trBtt|gt|}n t|g}t |dk}|s| dk rfdd| D}|St| }|ddg}t |t |}tt |t|}ddg} g} |D],} | D]} | | kr| | d} q| | q| S) a An utility function to get the key of the module to keep in full precision if any For example for CausalLM modules we may want to keep the lm_head in full precision for numerical stability reasons. For other architectures, we want to keep the tied weights of the model. The function will return a list of the keys of the modules to not convert in int8. Parameters: model (`torch.nn.Module`): Input model rNcs$g|]\}}t|tkr|qSr0)id)r4rHr*Z output_embr0r1 sz+get_keys_to_not_convert..r z.weightz.bias)rZ tie_weightsr rdictsumrAvalueskeysr@Zget_output_embeddingsZ named_modulesZnamed_parameterssetreplacer;) rEZ tied_modelZ tied_paramsZ tied_keysZhas_tied_paramsZlist_last_moduleZ list_modules intersectionZlist_untouchedZnames_to_removeZfiltered_module_namesrHZname_to_remover0rUr1get_keys_to_not_converts0       r_)NN)NNNF)NNN)importlib.metadatar!rMcopyr packagingrutilsrrrrrrZtorch.nnrZ pytorch_utilsrZ accelerater Zaccelerate.utilsr Z get_logger__name__rJr2rCrLrRrSr_r0r0r0r1s.        ` A *