U
0-e` @ s d dl Z d dlmZmZmZmZmZ d dlZd dlm Z ddl
m Z ddl m
Z
mZmZmZmZ G dd dZG dd d eZd"e jeed d d
Zd#e jdddZG dd deZd$ej jeeeejf eeeee f eee dddZ d%ej jeej eee eeeeeee f eee dddZ e jdddZ d&e jeeejeeejf f eeeeef f eeeeeeee f eee dd d Z G d d d eZ G d d! d!Z!dS )' N)DictListMappingOptionalUnion ) PartialState)PrefixedDataset find_devicenamed_module_tensorssend_to_deviceset_module_tensor_to_devicec @ s4 e Zd ZdZdZdd Zdd Zdd Zd d
Zd S ) ModelHooka
A hook that contains callbacks to be executed just before and after the forward method of a model. The difference
with PyTorch existing hooks is that they get passed along the kwargs.
Class attribute:
- **no_grad** (`bool`, *optional*, defaults to `False`) -- Whether or not to execute the actual forward pass under
the `torch.no_grad()` context manager.
Fc C s |S )z
To be executed when the hook is attached to the module.
Args:
module (`torch.nn.Module`): The module attached to this hook.
selfmoduler r Q/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/accelerate/hooks.py init_hook+ s zModelHook.init_hookc O s ||fS )a
To be executed just before the forward method of the model.
Args:
module (`torch.nn.Module`): The module whose forward pass will be executed just after this event.
args (`Tuple[Any]`): The positional arguments passed to the module.
kwargs (`Dict[Str, Any]`): The keyword arguments passed to the module.
Returns:
`Tuple[Tuple[Any], Dict[Str, Any]]`: A tuple with the treated `args` and `kwargs`.
r r r argskwargsr r r pre_forward4 s zModelHook.pre_forwardc C s |S )a:
To be executed just after the forward method of the model.
Args:
module (`torch.nn.Module`): The module whose forward pass been executed just before this event.
output (`Any`): The output of the module.
Returns:
`Any`: The processed `output`.
r )r r outputr r r post_forwardB s zModelHook.post_forwardc C s |S )z
To be executed when the hook is detached from a module.
Args:
module (`torch.nn.Module`): The module detached from this hook.
r r r r r detach_hookO s zModelHook.detach_hookN) __name__
__module__ __qualname____doc__no_gradr r r r r r r r r s
r c @ s8 e Zd ZdZdd Zdd Zdd Zdd Zd
d Zd S )
SequentialHookzX
A hook that can contain several hooks and iterates through them at each event.
c G s
|| _ d S N)hooks)r r# r r r __init__^ s zSequentialHook.__init__c C s | j D ]}||}q|S r" )r# r r r hookr r r r a s
zSequentialHook.init_hookc O s* | j D ]}|j|f||\}}q||fS r" )r# r )r r r r r& r r r r f s
zSequentialHook.pre_forwardc C s | j D ]}|||}q|S r" )r# r )r r r r& r r r r k s
zSequentialHook.post_forwardc C s | j D ]}||}q|S r" )r# r r% r r r r p s
zSequentialHook.detach_hookN) r r r r r$ r r r r r r r r r! Y s r! F)r r& appendc s |r,t dddk r, j}t t||}t drHt drH jn j _| | _t fdd}| _ S )a
Adds a hook to a given module. This will rewrite the `forward` method of the module to include the hook, to remove
this behavior and restore the original `forward` method, use `remove_hook_from_module`.
If the module already contains a hook, this will replace it with the new hook passed by default. To chain two hooks
together, pass `append=True`, so it chains the current and new hook into an instance of the `SequentialHook` class.
Args:
module (`torch.nn.Module`):
The module to attach a hook to.
hook (`ModelHook`):
The hook to attach.
append (`bool`, *optional*, defaults to `False`):
Whether the hook should be chained with an existing one (if module already contains a hook) or not.
Returns:
`torch.nn.Module`: The same module, with the hook attached (the module is modified in place, so the result can
be discarded).
_hf_hookN _old_forwardc sX j j f| |\} } j jr@t | |}W 5 Q R X n
| |} j |S r" )r( r r torchr )r r r r Z old_forwardr r new_forward s
z'add_hook_to_module..new_forward)
getattrr( remove_hook_from_moduler! hasattrr) forwardr functoolswraps)r r& r' Zold_hookr, r r+ r add_hook_to_modulev s
r3 )r c C s\ t | dr | j| t| d t | dr<| j| _t| d |rX| D ]}t|| qH| S )a
Removes any hook attached to a module via `add_hook_to_module`.
Args:
module (`torch.nn.Module`): The module to attach a hook to.
recurse (`bool`, **optional**): Whether to remove the hooks recursively
Returns:
`torch.nn.Module`: The same module, with the hook detached (the module is modified in place, so the result can
be discarded).
r( r) )r/ r( r delattrr) r0 childrenr. )r recursechildr r r r. s
r. c @ sx e Zd ZdZdeeeeej f e
e
ee e
e
eeee e f dddZ
dd Zd d
Zd d Zd
d Zdd ZdS )AlignDevicesHooka
A generic `ModelHook` that ensures inputs and model weights are on the same device for the forward pass of the
associated module, potentially offloading the weights after the forward pass.
Args:
execution_device (`torch.device`, *optional*):
The device on which inputs and model weights should be placed before the forward pass.
offload (`bool`, *optional*, defaults to `False`):
Whether or not the weights should be offloaded after the forward pass.
io_same_device (`bool`, *optional*, defaults to `False`):
Whether or not the output should be placed on the same device as the input was.
weights_map (`Mapping[str, torch.Tensor]`, *optional*):
When the model weights are offloaded, a (potentially lazy) map from param names to the tensor values.
offload_buffers (`bool`, *optional*, defaults to `False`):
Whether or not to include the associated module's buffers when offloading.
place_submodules (`bool`, *optional*, defaults to `False`):
Whether to place the submodules on `execution_device` during the `init_hook` event.
NF)execution_deviceoffloadio_same_device weights_mapoffload_buffersplace_submodules skip_keysc C s@ || _ || _|| _|| _|| _|| _|| _d | _i | _i | _ d S r" )
r9 r: r; r< r= r> r? input_deviceZparam_original_devicesZbuffer_original_devices)r r9 r: r; r< r= r> r? r r r r$ s
zAlignDevicesHook.__init__c
C s: d| j d| j d| j d| j d| j dt| j d
S )Nz"AlignDevicesHook(execution_device=z
, offload=z, io_same_device=z, offload_buffers=z, place_submodules=z , skip_keys=))r9 r: r; r= r> reprr? r r r r __repr__ s 8zAlignDevicesHook.__repr__c C s | j s8| jd k r8t|| jdD ]\}}t||| j q n| j rdd t|| jdD | _| jd krdd t|| j| jdD | _t|| j| jdD ]\}}t||d q| js| jd k r|j| jdD ]\}}t||| j q|S )N)r6 c S s i | ]\}}||j qS r )device.0nameparamr r r
s z.AlignDevicesHook.init_hook..c S s i | ]\}}|| d qS )cputorF r r r rJ s Zinclude_buffersr6 meta) r: r9 r r> r
original_devicesr< r= Z
named_buffers)r r rH _r r r r s2
zAlignDevicesHook.init_hookc O s | j rt||g| _| jrt|| j| jdD ]h\}}d }d|krz|dd| j krz| j| j
t j krz| j|dd }t
||| j| j| |d q,t|| jt|| j| jdfS )NrN weightSCB)valuefp16_statisticsr? )r; r
r@ r: r r= r> replacer< keysZdtyper* Zint8r
r9 r r? )r r r r rH rQ rU r r r r s0 z AlignDevicesHook.pre_forwardc C sr | j rLt|| j| jdD ]2\}}t||d t|jdkrd |j_d |j_ q| j
rn| j d k rnt || j | j
d}|S )NrN rO Z Linear8bitLtrV )r: r r= r> r
typer staterS ZCxBr; r@ r r? )r r r rH rQ r r r r & s
z AlignDevicesHook.post_forwardc C sF | j rB| j D ]0\}}|tdkrt|||| j|d d q|S )NrO )rT )r: rP itemsr* rE r
r< get)r r rH rE r r r r 5 s
z AlignDevicesHook.detach_hook)NFFNFFN)r r r r r r intstrr* rE boolr r r$ rD r r r r r r r r r8 s, r8 )r r9 r? preload_module_classesc C s` t | ds,t| dkr,t| t||d |dk rD| jj|krDdS | D ]}t|| qLdS )a!
Recursively attaches `AlignDevicesHook` to all submodules of a given model to make sure they have the right
execution device
Args:
module (`torch.nn.Module`):
The module where we want to attach the hooks.
execution_device (`int`, `str` or `torch.device`):
The device on which inputs and model weights should be placed before the forward pass.
skip_keys (`str` or `List[str]`, *optional*):
A list of keys to ignore when moving inputs or outputs between devices.
preload_module_classes (`List[str]`, *optional*):
A list of classes whose instances should load all their weights (even in the submodules) at the beginning
of the forward. This should only be used for classes that have submodules which are registered but not
called directly during the forward, for instance if a `dense` linear layer is registered, but at forward,
`dense.weight` and `dense.bias` are used in some operations instead of calling `dense` directly.
r( r rV N) r/ lenZ
state_dictr3 r8 __class__r r5 attach_execution_device_hook)r r9 r? r` r7 r r r rc = s rc )r r9 r: r< r= module_namer? r` c C s t | }|o |dk o | jj|k} tt|dks4| r|dk rbt|dkrR| dnd}
t||
} nd} t||| || |d} t| | dd | rdS | D ]>\}
}t|dkr| d|
n|
}
t ||||||
||d qdS ) aP
Recursively attaches `AlignDevicesHook` to all submodules of a given model that have direct parameters and/or
buffers.
Args:
module (`torch.nn.Module`):
The module where we want to attach the hooks.
execution_device (`torch.device`, *optional*):
The device on which inputs and model weights should be placed before the forward pass.
offload (`bool`, *optional*, defaults to `False`):
Whether or not the weights should be offloaded after the forward pass.
weights_map (`Mapping[str, torch.Tensor]`, *optional*):
When the model weights are offloaded, a (potentially lazy) map from param names to the tensor values.
offload_buffers (`bool`, *optional*, defaults to `False`):
Whether or not to include the associated module's buffers when offloading.
module_name (`str`, *optional*, defaults to `""`):
The name of the module.
skip_keys (`str` or `List[str]`, *optional*):
A list of keys to ignore when moving inputs or outputs between devices.
preload_module_classes (`List[str]`, *optional*):
A list of classes whose instances should load all their weights (even in the submodules) at the beginning
of the forward. This should only be used for classes that have submodules which are registered but not
called directly during the forward, for instance if a `dense` linear layer is registered, but at forward,
`dense.weight` and `dense.bias` are used in some operations instead of calling `dense` directly.
Nr .rd )r9 r: r< r= r> r? T)r' r9 r: r< r= re r` r? )
r rb r ra listr r8 r3 named_childrenattach_align_device_hook)r r9 r: r< r= re r? r` ZdirectsZ full_offloadprefixZprefixed_weights_mapr&
child_namer7 r r r rj _ s> $ rj c C s" t | | D ] }t| qdS )z
Recursively removes all hooks attached on the submodules of a given model.
Args:
module (`torch.nn.Module`): The module on which to remove all hooks.
N)r. r5 remove_hook_from_submodules)r r7 r r r rm s rm c s t tsNt tsNs4t d|dd}t| | nt| d||||d dS t tsn fdd D t tsfdd D | kr|krڈ| st | ||dkd|d }t| | t| | n| krL|krLt| | d|||||d
t| d s6t | |dk|d }t| | t| | ||d
n(|dkrtt dd|d }t| | |
D ]B\} }
t |dkr| d| n| } t |
||| ||d q|dS )a
Attaches `AlignDevicesHook` to all blocks of a given model as needed.
Args:
module (`torch.nn.Module`):
The module where we want to attach the hooks.
execution_device (`torch.device` or `Dict[str, torch.device]`, *optional*):
The device on which inputs and model weights should be placed before the forward pass. It can be one device
for the whole module, or a dictionary mapping module name to device.
offload (`bool`, *optional*, defaults to `False`):
Whether or not the weights should be offloaded after the forward pass. It can be one boolean for the whole
module, or a dictionary mapping module name to boolean.
weights_map (`Mapping[str, torch.Tensor]`, *optional*):
When the model weights are offloaded, a (potentially lazy) map from param names to the tensor values.
offload_buffers (`bool`, *optional*, defaults to `False`):
Whether or not to include the associated module's buffers when offloading.
module_name (`str`, *optional*, defaults to `""`):
The name of the module.
skip_keys (`str` or `List[str]`, *optional*):
A list of keys to ignore when moving inputs or outputs between devices.
preload_module_classes (`List[str]`, *optional*):
A list of classes whose instances should load all their weights (even in the submodules) at the beginning
of the forward. This should only be used for classes that have submodules which are registered but not
called directly during the forward, for instance if a `dense` linear layer is registered, but at forward,
`dense.weight` and `dense.bias` are used in some operations instead of calling `dense` directly.
T)r9 r; r? r> )r9 r: r< r= re r? Nc s i | ]
}| qS r r rG key)r9 r r rJ s z6attach_align_device_hook_on_blocks..c s i | ]
}| qS r r rn )r: r r rJ s rd )r9 r= r; r> r? )r9 r: r< r= re r? r` r( )r9 r; r? )r` r? r rf rg )
isinstancer dictr8 r3 rj rX rc r/ r\ ri ra "attach_align_device_hook_on_blocks) r r9 r: r< r= re r? r` r& rl r7 r )r9 r: r rr s %
rr c @ sF e Zd ZdZd eeeeej f ed dddZ
dd Z d d
Z dS )
CpuOffloada
Offloads a model on the CPU until its forward pass is called. The model will not be offloaded back to the CPU after
the forward, the user needs to call the `init_hook` method again for this.
Args:
execution_device(`str`, `int` or `torch.device`, *optional*):
The device on which the model should be executed. Will default to the MPS device if it's available, then
GPU 0 if there is a GPU, and finally to the CPU.
prev_module_hook (`UserCpuOffloadHook`, *optional*):
The hook sent back by [`cpu_offload_with_hook`] for a previous model in the pipeline you are running. If
passed, its offload method will be called just before the forward of the model to which this hook is
attached.
NUserCpuOffloadHook)r9 prev_module_hookc C s || _ |d k r|nt j| _d S r" )ru r Zdefault_devicer9 )r r9 ru r r r r$ 4 s zCpuOffload.__init__c C s
| dS )NrK rL r r r r r = s zCpuOffload.init_hookc O s8 | j d k r| j || j t|| jt|| jfS r" )ru r: rM r9 r r r r r r @ s
zCpuOffload.pre_forward)NN)
r r r r r r r^ r] r* rE r$ r r r r r r rs % s rs c @ s( e Zd ZdZdd Zdd Zdd ZdS ) rt z
A simple hook grouping a model and a `ModelHook`, which provides easy APIs for to call the init method of the hook
or remove it entirely.
c C s || _ || _d S r" )modelr& )r rv r& r r r r$ M s zUserCpuOffloadHook.__init__c C s | j | j d S r" )r& r rv rC r r r r: Q s zUserCpuOffloadHook.offloadc C s t | j d S r" )r. rv rC r r r removeT s zUserCpuOffloadHook.removeN)r r r r r$ r: rw r r r r rt G s rt )F)F)NN)NFNFrd NN)NFNFrd NN)"r1 typingr r r r r r* Ztorch.nnnnrZ r utilsr r
r r r
r r! Moduler_ r3 r. r8 r] r^ rE rc rj rm rr rs rt r r r r sn : 6 x
$
L
n"