U *-e@sddlZddlZddlZddlZddlZddlmZmZddlm Z m Z ddl m Z m Z mZmZmZmZmZmZmZmZmZmZmZmZddlZddlmZddlmZddl mm!Z"ddlm#Z#ddl$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*ddl+m,Z,m-Z-m.Z.ddl/m0Z0d d l1m2Z2m3Z3d d d dddgZ4e5e6Z7dZ8dZ9dZ:dZ;GdddeZfZ?esz/FlatParameter._init_metadata..cSsh|] }|jqSr1r{r}Zspir1r1r2rsNcSsg|]}dqS)Fr1r}_r1r1r2 sz0FlatParameter._init_metadata..cSsg|]}dqSNr1rr1r1r2rsF)lenrjrTrUrVrWrXrizipappendtuplerZrYr\unionr]rerfrrrangerhrgsizerQ_post_backward_called)rmrNrqrrrsrtrurvrwrxryZnumels_without_paddingnumel is_paddingparamr1r1r2_init_metadata]sR         zFlatParameter._init_metadata)NT)r)r*r+r7rHrIr9rFrr%r8rrr?r&rr:r;rr rkrErl classmethodrr1r1r1r2r"sP x        ) metaclassc s eZdZdZeeejefej e j e e ee jee je eje d fdd ZddZe ddd Zeeeejfej ee d d d d ZeeeejfedddZeeeedddZeeee edddZee jee jd dddZe ddZeeed dddZ eeee!dfd d!d"Z"e#eeeeeefd#d$d%Z$e#eeeeeefd#d&d'Z%e#eeee j&d#d(d)Z'eeeefd*d+d,Z(e)e*d*d-d.Z+e)e d d*d/d0Z,e d*d1d2Z-d3d4Z.d5d6Z/e d*d7d8Z0d9d:Z1e jd*d;d<Z2eed=d>d?Z3e jd d=d@dAZ4dBdCZ5dDdEZ6e dFdGZ7dHdIZ8dJdKZ9dLdMZ:e;jdSdTZ?dUdVZ@d d*dWdXZAe)dee jeBedYdZd[ZCe)deeeedYd\d]ZDe)e d d^d_d`ZEe)d d*dadbZFe;jr< numel_to_padpadding_tensorZ transform_t extensionfqnr1r1r2r0s                     z-FlatParamHandle._init_flat_param_and_metadatatensorsrzcCsd}d}d}|D]}t|tr&td|dkr>|s>td|dk rf|j|krftd|d|j|js|dk r|j|krtd|dk r|j|krtd|d|j|j}|p|j}|j}q|dk std|||fS) zV Validates the tensors to flatten and returns any necessary metadata. Nz Cannot flatten a `FlatParameter`z$Cannot flatten integer dtype tensorsz0Must flatten tensors with uniform dtype but got z and zNMust flatten tensors with uniform `requires_grad` when `use_orig_params=False`z5Must flatten tensors on the same device but got both z!Requires non-empty `tensors` list) rLr"rZis_floating_pointrrrorrj)rNrrrrtensorr1r1r2rs<  z,FlatParamHandle._validate_tensors_to_flatten)rrrzc Cst|dkrtd|dkr*td|||\}}}g}|dkrd}|D]`}|||} | dkr| |krt| |d|} || || 7}|tt|||7}qN|j ||j } | dkr| |j krt| |d|} || || 7}ndd|D}tj |ddS)a  Flattens ``tensors`` into a single flat tensor optionally including padding if ``aligned_numel`` is greater than 0, where ``aligned_numel`` gives the numel required to have address alignment. NOTE: The padding alignment algorithm must be kept in sync with :meth:`_init_flat_param_metadata`. We separate the two methods because the initialization happens once, whereas this method may be called multiple times throughout training (e.g. for checkpointing). rzExpects non-empty `tensors`rFcSsg|]}tt|qSr1)rHflatten_detach_if_needed)r}rr1r1r2rsz3FlatParamHandle.flatten_tensors..dim) rrrrrrHrrrrcat) rNrrrrrZ flat_tensorsrrrrr1r1r2flatten_tensorssJ     zFlatParamHandle.flatten_tensors)rrrorzcCs|||}t||dS)N)ro)rr")rNrrroZflat_param_datar1r1r2rs z/FlatParamHandle.flatten_tensors_into_flat_param)rrrzcCsh|dk |_|dk |_|jr0|js0||_|j|_n|p8|j|_|pD|j|_|jdk sVt|jdk sdtdS)a0 Precondition: ``self.flat_param`` is set. This ensures that this handle's parameters have a single dtype. Postcondition: This sets ``self._fwd_bwd_param_dtype`` and ``self._reduce_dtype``. If ``mp_param_dtype`` or ``mp_reduce_dtype`` is ``None``, then we assume the original parameter dtype. One special case is if ``mp_param_dtype`` is not ``None`` and ``mp_reduce_dtype`` is ``None``, in which case we assume the gradient reduction dtype matches the forward/backward parameter dtype. N)_low_prec_param_dtype_specified _low_prec_reduce_dtype_specifiedr _reduce_dtyperrj)rNrrr1r1r2r#s     z)FlatParamHandle._init_param_reduce_dtypescCs|j}|js$|dd|dnt|dkd|}t||j |j \}}| |||j }||j dd}||||| dkr| d|jr|dS)aE Shards the handle's ``FlatParameter``. This allocates new memory for the sharded flat parameter and frees the unsharded flat parameter's storage. Postcondition: ``self.flat_param`` is the sharded flat parameter. Shard metadata attributes are set for all sharding strategies. rrz;The `FlatParameter` is not the sole occupant of its storageN)ruses_sharded_strategy_init_shard_metadatarrZstorage_offset_typed_storager# _get_shardrrset__sizeZ_resize_r_use_sharded_views)rNrZ orig_storagesharded_flat_param numel_paddedZ start_idxZend_idxr1r1r2shardGs*     zFlatParamHandle.shard)runsharded_start_idxunsharded_end_idxrzcCs|j}||_|}t|dko(||kd|d|t||kd|d||||}t||jkstd|jdt|||_ ||_ dS) a Initializes shard-related metadata for this rank's shard of the flat parameter: ``_sharded_size``, ``_shard_param_infos``, and ``_shard_numel_padded``. Args: numel_padded (int): Numel padded for this rank's sharded flat parameter. unsharded_start_idx (int): Start index in the unsharded flat parameter assigned to this rank. unsharded_end_idx (int): End index (inclusive) in the unsharded flat parameter assigned to this rank. Precondition: ``self.flat_param`` 's data is the sharded flat parameter. rzunsharded_start_idx: z unsharded_end_idx: znumel_padded: z sharded_flat_param_numel: zExpects length but got N) rrrSrr_get_shard_metadatarrTrjr[r^)rNrrrrsharded_flat_param_numelshard_param_infosr1r1r2rfs*  z$FlatParamHandle._init_shard_metadata.)rrrzcCs |}t|t|jjks.)rrrrYr)rNZcumulative_sumZstartsZendsrGr1r1r2rs z'FlatParamHandle._get_flat_param_offsetsc Csg}g}g}g}t|jj|jj|jj|jjD]D\}}}}|jsBq.||||||||j|j fq.t t |t |t ||S)z Returns shard-related metadata specific to this rank's shard of the flat parameter. NOTE: The returned tuple does not include elements for alignment padding but does account for the padding. ) rrrWrVrZr[r@rrCrDr$r) rNZ fqns_listZ shapes_listZ numels_listZshard_param_offsetsrrrrr1r1r2shard_metadatas4    zFlatParamHandle.shard_metadatacCsH|j}|j|jkr<|js |j|_|js4|js4|j|_|j|_td}|j rft |j|kd|jn | |j|j |_ |j r|j |_ tj|j |d|_|jrtj|j |j|jd|_t|j|jrD|jr|jn|j}||j}tj||j|d|_|j|_t|j|jrDtj||j|jd|_t|jdS)a This initializes some attributes on the handle's ``FlatParameter``. This should be called during lazy initialization since it requires the parameter to be on the compute device if not offloading to CPU and we want to give users the chance to move the parameter appropriately after the FSDP constructor. For each tensor attribute on the ``FlatParameter``, see the unshard and reshard methods in this class for the allocation and free pattern. cpuzWExpects the `FlatParameter` to be on CPU when parameter CPU offloading is enabled, not r)rrN)rrrrrrrrHrrr_check_on_compute_devicernr_Z pin_memory zeros_likerc_uses_param_mixed_precision empty_likerbrrrremptyr`rrRra)rNrZ cpu_deviceZunsharded_param_dtypeZpadded_unsharded_numelr1r1r2init_flat_param_attributes@sf          z*FlatParamHandle.init_flat_param_attributescCs|jtjkr|jr|d}|jr2|js2|}|jrH|j sH| sHnB|j rb|j sb| d}n(|j r|jj|jkr|j|jddd}||j|S)a$ Returns: ``False`` if this is a no-op and ``True`` otherwise. Postcondition: ``self.flat_param`` 's data is on the device for communication and is what should be all-gathered. This means that it matches the dtype of the expected unsharded parameter. FTZ non_blocking)rrSUMMON_FULL_PARAMS_skipped_use_sharded_viewsrrr_writeback_orig_paramsrr needs_unshardr_force_full_precision_use_low_precision_shardrr flat_param_tor)rNretr1r1r2 pre_unshards.    zFlatParamHandle.pre_unshardcCsF||j}t|j|j|j|jj|jdd|j|_ dS)z Allocates the low precision shard directly on the compute device and switches to using the low precision sharded flat parameter. TrN) _check_low_precision_shardrrrbr_rcopy_torrn)rNrr1r1r2rsz(FlatParamHandle._use_low_precision_shardcCsJ|s*|jr|n|j}||dS|}||}||dS)a Runs the unshard logic. This includes all-gathering the flat parameter and switching to using the unsharded flat parameter. If the handle does not need unsharding, then this only switches to using the unsharded flat parameter. For ``NO_SHARD``, this is a no-op. If FSDP is in :meth:`summon_full_params` and the handle uses parameter mixed precision, then the parameter is forced to full precision. N)rr _get_padded_unsharded_flat_paramr_use_unsharded_flat_param"_alloc_padded_unsharded_flat_param_all_gather_flat_param)rNunsharded_flat_parampadded_unsharded_flat_paramr1r1r2unshards    zFlatParamHandle.unshardcCs,|js dS|}||k}| S)z=Returns if the handle's flat parameter needs to be unsharded.F)rr%rrr)rNr)Zalready_unshardedr1r1r2rs zFlatParamHandle.needs_unshardcCs0||j}|}||t||j|S)a  Allocates the *padded* unsharded flat parameter. The unpadded unsharded flat parameter is always a view into the padded one. This padded parameter is saved to a different attribute on the ``FlatParameter`` depending on if we force full precision. )_check_sharded_strategyrr%_check_storage_freedrrRrNrr)r1r1r2r's   z2FlatParamHandle._alloc_padded_unsharded_flat_paramcCsb||j}|jrX|jrX|j}t|j|jkd|j|j dkr^t |jn|j}|S)z Returns a reference to the padded unsharded flat parameter depending on the calling context. This should only be called if using a sharded strategy. zExpects full precision but got r) r,rrrrarrrr`untyped_storagerrr.r1r1r2r%s     z0FlatParamHandle._get_padded_unsharded_flat_param)r*rzcCstt|dot|dd|jj}||j}t||kd|d||jrtt |t |j }t j |||j d}nt |||j |S)z All-gathers the handle's flat parameter to the destination ``padded_unsharded_flat_param``, and switches to using the all-gathered tensor. rrzEExpects a process group and world size to have been set via `shard()`zExpects z numel but got group)rhasattrrrnrrZis_cpurrHrrZget_world_sizerZ all_gatherall_gather_into_tensor)rNr*rZexpected_numelZ tensor_listZworkr1r1r2r(s8   z&FlatParamHandle._all_gather_flat_paramcCsx|jj}|d|||j_|jtjk}|jtjk}|j rd|j rN|rNdS|j | o\| dn|rt|j dddS)z Switches to using the *unpadded* unsharded flat parameter, which is a view into the *padded* unsharded flat parameter. NrF) rrQrviewrnrrFORWARD BACKWARD_PRErrr)rNr*unsharded_size in_forwardZin_pre_backwardr1r1r2r&Bs     z)FlatParamHandle._use_unsharded_flat_paramcCs$|jr|jr|||jdS)zo Runs the post-unshard logic. This includes freeing the low precision shard if needed. N)rr!_free_low_precision_sharded_paramrrrNr1r1r2 post_unshardbs zFlatParamHandle.post_unshardcCs,|t|jj|jt|jjdS)z/Frees the low precision sharded flat parameter.N)r"rrrbrcurrent_streamrr:r1r1r2r9ks z1FlatParamHandle._free_low_precision_sharded_paramcCs"|js|dS|j}||tjdtj|jd}|jdk|d<t j ||j d|d|j krtd|_ |dStj|j|jd}|jdkr|jt jjkrtd|jdd|_ tj|j|jd}n||j|j|_ |j }t |||j |jj}|d|||_|dS) a Unshards the handle's ``FlatParameter`` 's gradient. If all ranks have ``None`` gradient, then all original parameters will as well. This method performs an all-reduce and an all-gather. The additional all-reduce is tolerable since this method is not meant to be used on the computation critical path. Postcondition: ``_saved_grad_shard`` is defined and contains the value to set ``flat_param.grad`` after gradients are resharded. Nr)rrrr0r[Rank z] Only some but not all ranks have a `None` `FlatParameter` gradient, so FSDP is using zeros to approximate those ranks' sharded gradients being `None`)r_use_unsharded_grad_viewsr_check_unshardedrHZzerosZint32rgradrZ all_reducerrrdrrRr DebugLevelINFOwarningswarnrrS_check_shardedr3rQrr4)rNrZ num_grad_noneZpadded_unsharded_grad sharded_gradr7r1r1r2 unshard_gradzsH     zFlatParamHandle.unshard_gradcCs4|jr||jsdS|jj|j_t|jddS)Nrd)r_use_sharded_grad_viewsrrrdr@delattrr:r1r1r2 reshard_grads  zFlatParamHandle.reshard_gradcCs(t|jtjtjfkd|j}|jdk r$|j|jksJ|jj |j kr$| |j|jj |j k}t| pp|j d|j d|jj |j|j k}|r|s|jj |_|j}ntt|dd|j}|j j}|jr|j|kr|||_ n,|j}t|j|kd|d|jd|_dS) z Prepares the gradient for the backward computation by saving and clearing any existing sharded gradient in ``.grad`` to enable computing a new unsharded gradient. z:Expects to be in `BACKWARD_PRE` or `IDLE` (if prefetching)Nz&Expects the sharded gradient to be on rrcz7`_cpu_grad` should be defined if the gradient is on CPUzFExpects `.grad` to be the unsharded gradient in `no_sync()` with size z but got size )rrrr6rrr@rrQrrrr_rnrdr2rcrrr$rR)rNrZgrad_offloadedZprev_iter_synced_gradientsrFZlocal_shard_dtypeZpadded_unsharded_sizer1r1r2prepare_gradient_for_backwardsV       z-FlatParamHandle.prepare_gradient_for_backwardcsfdd}j}t|drB|||j|_||nlt|dr|||jdk rv|j|jr|j|_|jdk r||nt j p|j dt|drt |ddS)z Prepares the gradient for optimizer computation by moving the sharded gradient to the ``.grad`` attribute. csNjsJjrJt|jdk d|jjjkrJ|jj|j_jrJ dS)NzUnexpected None grad!) rrrr@rrr$rnrrH)rr:r1r2"cast_grad_to_param_dtype_if_neededs  zVFlatParamHandle.prepare_gradient_for_optim..cast_grad_to_param_dtype_if_neededrcrdNzcAll sharded parameters that received a gradient in the post-backward should use `_saved_grad_shard`) rr2rE _check_on_cpurcr@rrdrrrrI)rNrLrr1r:r2prepare_gradient_for_optims.             z*FlatParamHandle.prepare_gradient_for_optimc cs|t|j|jjkd|jjd|j||j|j}|}t||kd| t d| z dVW5t|j|jjkd|jjd|j| }|d|j|j||XdS)a[ Moves the unpadded unsharded flat parameter to CPU while in the context and moves it back to the previous device upon exit. For now, this assumes the ``FlatParameter`` is the unpadded unsharded flat parameter since (1) there is no reason to include the padding in the copy and (2) there is no use case for the sharded flat parameter. Precondition: ``self.flat_param`` 's data is the unpadded unsharded flat parameter on the compute device, and the handle uses a sharded strategy. Postcondition: Same as the precondition. z Expects size rzEExpects the unpadded parameter to be a view into the padded parameterrN)r,rrrrQrrZ _data_ptrr%rrHr_free_unsharded_flat_paramr'rr#r&)rNZunpadded_storage_ptrZpadded_storage_ptrr*r1r1r2to_cpu)s4  zFlatParamHandle.to_cpu)free_unsharded_flat_paramcCs||r|dS)av Runs the reshard logic. This includes freeing the unsharded flat parameter if ``free_unsharded_flat_param`` and switching to using the sharded flat parameter. Note that this also implicitly offloads the sharded flat parameter (if CPU offload is enabled) by pointing it to the ``_local_shard`` attribute which resides on CPU. N)_use_sharded_flat_paramrO)rNrQr1r1r2reshardYs zFlatParamHandle.reshardcCs|jr|js|js|dS)a; Runs the post-reshard logic. This includes freeing any memory that can now be freed given that the ``FlatParameter`` points to the full precision sharded flat parameter. Precondition: ``self.flat_param`` 's data points to the full precision sharded flat parameter. N)rrrr9r:r1r1r2 post_reshardis zFlatParamHandle.post_reshardcCs@||}||||t||jt|dS)z Frees the padded unsharded flat parameter. The tensor to free depends on the calling context since the unshard may have forced full precision, in which case a different tensor is used. N)r,r%_check_storage_allocatedrrrr<r)rNr)r1r1r2rO|s  z*FlatParamHandle._free_unsharded_flat_paramcCs|j}|jr8|jtjk}to,|o,|jtk}|r8|j }|j r`|j j }t |t dkd||j |_ |jr|rz||_n||r|js|jdk o|jo|jj|jk}|r|n|dS)z-Switches to using the sharded flat parameter.rz-Expects the local shard to be on CPU but got N)rrrrr5rHZis_grad_enabledr*NO_RESHARD_AFTER_FORWARD_HANDLE_STRATEGIESrnrr_rrrrrr@rrrQr>rH)rNrr8Zskip_use_sharded_viewsr)rZaccumulated_grad_in_no_syncr1r1r2rRsF     z'FlatParamHandle._use_sharded_flat_param)rrzcCs>|j}|dkr|}ddttj||jdd|j|jD}|S)a2 Returns unflattened ``Tensor`` views into ``tensor`` if it is not ``None`` or ``flat_param`` otherwise, where the unflattening is based on ``flat_param`` 's metadata. Examples for ``tensor`` include ``flat_param.grad`` or unsharded tensor optimizer state. Ncss$|]\}}}t|||VqdSr)r r4)r}Z subtensorrZparam_extensionr1r1r2 sz>FlatParamHandle._get_unflat_views_unaligned..rr)rrrHsplitrZrVrX)rNrrviewsr1r1r2rs z+FlatParamHandle._get_unflat_views_unalignedcCsv|j}|dkr|}tj||jdd}d}g}t||jD]8\}}|rFq8|t||j ||j ||d7}q8|S)z This has the same contract as :meth:`_get_unflat_views_unaligned` except it checks for ``None`` placeholders representing padding for alignment, which may incur slightly more CPU overhead. Nrrr) rrHrXrYrrirr r4rVrX)rNrrZsplitsidxrYrXrr1r1r2rs*  z)FlatParamHandle._get_unflat_views_aligned)rrzcCs|j}|||}ddlm}tt||jD]\}\}\}}} |jr|rt ||krr| ||t |q4|jj |} | ||| || _q4|r| ||t |q4|} |jr|jtjkr||jj|<n"|jtjkr|jj|} || _| } |||| |jr4|jtjkr4| |j|<q4t|jjD]\}\}}} } }} t|| }t| p^t|t j d|dt ||jr|r|jj|}| |||||_nD|r| |||n.|||||jr,|jtjkr,||j|<q,dS)a Unflattens the unsharded flat parameter by setting the original parameter variables to be views into it. Args: as_params (bool): If ``True``, then registers the original parameters as ``nn.Parameter`` s; if ``False``, then registers the original parameters only as ``Tensor`` s. ``False`` should be used during forward/backward computation and when hiding the original parameters from :meth:`nn.Module.named_parameters`. r)DTensorz as_params=z type(prim_param)=N)rr?rZtorch.distributed._tensorr[rrrUrtyperr:rkrernrrr5rgr6r _parametersr\rMrrLrf)rNrrrYr[rr4r3r4rrZ param_varrr<r= prim_paramZ shared_paramr1r1r2rst               z$FlatParamHandle._use_unsharded_viewsc Cs|jjdkr.t|jj|jjD] }d|_qdS||jj||jj}tt||jj D]\}\}\}}}t t |||jj |dt ||}|j|jks|j|jks|j|jkr|jdkrt||_||j_q\||_q\t|jjD]\}\}}}} } }t t |||r|d|n|dt ||}t | | } |j| jjksn|j| jjksn|j| jjkr|jdkrt||_| j|j_q| j|_qdS)z Unflattens the unsharded flat parameter's gradient by setting the original parameter variables' gradients to be views into it. Nz is missingr)rr@rrerfr?rrrrUrr2rWrMrrrrHrrnr\) rNrrYrr4r3r4rr5r<r=r^r1r1r2r>Is`                  z)FlatParamHandle._use_unsharded_grad_viewsc cs*|jddz dVW5|jddXdS)a" Assumes the flat parameter is unsharded. When in the context, unflattens the original parameters as ``nn.Parameter`` views into the flat parameter, and after the context, restores the original parameters as ``Tensor`` views into the flat parameter. TrFN)rr:r1r1r2unflatten_as_paramss  z#FlatParamHandle.unflatten_as_paramscCs8d|_|js|jdddS|j}||tjd|jj|jjdd}t |j |j |j D]J\}}\}}}| ||||js||_qX|j}|j} |||| |_qX|jjdk sttt |jj|jjD]6\} \}\}}}} } }| |||t| | } | |_q|jtjkr4tt|jjD]} d|jj| <q dS)a5 Sets the original parameter variables' data to be flattened views into the sharded flat parameter. The views are kept as flattened to simplify the case where a parameter is sharded across ranks. Parameters whose data is not present in the sharded flat parameter have their data set to a size-0 empty tensor. We do not delete them to ensure to preserve expected behaviors like model printability. Parameters whose data is present must preserve their variables to be passable to an optimizer. NTrrF)rrro)rrrrrErHrrrrrer[rUrr@rnrArBrfrjrr\rMrr BACKWARD_POSTrrrg)rNrZsize_0_empty_tensorrrr3r4roffsetrBrr<r=r^r1r1r2rsH    z"FlatParamHandle._use_sharded_viewscCsb|j}|||j}|dkr|jj|||j_|jr:||jr.|n |jdddS)z.cSsh|] }|jqSr1r{rr1r1r2r s)rrUrr\r:r1r1r2 _get_modules szFlatParamHandle._get_modulescCs*t|jdr|jsdS|jj}||kS)z Returns if ``tensor`` is *currently* sharded. For ``NO_SHARD``, we choose to have this always return ``False`` for clarity. rSF)r2rrrSr)rNr sharded_sizer1r1r2rb s zFlatParamHandle.is_shardedccs>dd|jjD}t|jj|D]}|\}}}||fVq dS)NcSs$g|]\}}}}}}t|||qSr1r%r}r3r4r5rr1r1r2r) sz6FlatParamHandle.param_module_names..)rr\rrU)rNrurlr3rr5r1r1r2param_module_names( s    z"FlatParamHandle.param_module_namesccs,dd|jjDD]\}}}||fVqdS)NcSs$g|]\}}}}}}t|||qSr1rrrsr1r1r2r9 sz=FlatParamHandle.shared_param_module_names..)rr\)rNr3rr5r1r1r2shared_param_module_names8 s  z)FlatParamHandle.shared_param_module_namescCs4g}t|jj|jjD]\}}|jr||q|S)z@Returns the FQNs of the parameters present in this rank's shard.)rrrWr[r@r)rNZ fqns_in_shardrrr1r1r2_fqns_in_shardF s  zFlatParamHandle._fqns_in_shardcCs^|j}t|dr|j}nBt|dr*|j}n0t|jdkpN|j pN|jtj tj fkd|j}|S)z&Returns the handle's sharded gradient.rcrdNzZSharded strategies should use `_cpu_grad` or `_saved_grad_shard` unless in IDLE or FORWARD) rr2rcrdrr@rrrr5r)rNrr@r1r1r2rFQ s     zFlatParamHandle.sharded_gradcCsf|js dSt|jtjkd|j}|jdk s0tt|jD]&\}}|j r:|j dk sVtd|j |<q:dS)a* Resets ``_is_grad_none_mask`` as needed. This method should only be called in the post-backward after gradient computation, in which case if a parameter requires gradient, then it will surely receive a gradient and we may reset its mask entry to ``False``. NzIExpects to only be called in the post-backward after gradient computationF) rrrrr`rrerjrrorh)rNrrrr1r1r2_reset_is_grad_nonep s z#FlatParamHandle._reset_is_grad_nonecCst|jddS)NzExpects sharded strategy)rrr:r1r1r2r, sz'FlatParamHandle._check_sharded_strategy)rcCst|j|jkd|jdS)Nz+Expects tensor to be on the compute device )rrrNrr1r1r2r s  z(FlatParamHandle._check_on_compute_devicecCs"t|jtdkd|jdS)Nrz$Expects tensor to be on CPU but got )rrrHrxr1r1r2rM s zFlatParamHandle._check_on_cpucCs$|}t|dkd|dS)Nrz6Expects storage to be freed but got storage with size rrrrZ storage_sizer1r1r2r- s  z$FlatParamHandle._check_storage_freedcCs|}t|dkddS)NrzExpects storage to be allocatedryrzr1r1r2rU s z(FlatParamHandle._check_storage_allocatedcCsPt|jdtt|jdddk d|jjj}t||jkd|jd|dS)Nz&Not using low precision for parametersrbzExpects `_mp_shard` to existz)Expects the low precision shard to be on r)rrrMrrbr)rNrr1r1r2r" s z*FlatParamHandle._check_low_precision_shardcCsHd}t|dk |d|jj}t||k|d|d|dS)NzExpects tensor to be unsharded but got `None` with size r)rrrQr)rNr msg_prefixr7r1r1r2r? s z FlatParamHandle._check_unshardedcCsHd}t|dk |d|jj}t||k|d|d|dS)NzExpects tensor to be sharded r{r|r)rrrSr)rNrr}rqr1r1r2rE s zFlatParamHandle._check_shardedcCs |jtjkSr)rr'r.r:r1r1r2r sz%FlatParamHandle.uses_sharded_strategycCs |j|jkSr)rrr:r1r1r2r sz+FlatParamHandle._uses_param_mixed_precisioncCs |j|jkSr)rrr:r1r1r2_uses_reduce_mixed_precision sz,FlatParamHandle._uses_reduce_mixed_precisioncCs(|js |jo&|jtjkp&|jj o&|jSr)rr~rrrrZtrainingrr:r1r1r2r s   z%FlatParamHandle._force_full_precisioncCs |jdk S)a> This property is used for sharding strategies that do not free after forward with ``use_orig_params=True``. This returns if this handle is currently in a state where it has skipped using sharded views, in which case it can restore view invariants via ``_use_sharded_views()``. N)rr:r1r1r2r sz*FlatParamHandle._skipped_use_sharded_views)N)N)hr)r*r+r7rrr:rkrr;rHrr'rErrrZ ProcessGrouprrrr rFrrrrr"rrZno_gradrrr?r staticmethodrrrIr rrr$rrr!rr+rr'r%r(r&r;r9rGrJrKrN contextlibcontextmanagerrPrSrTrOrRr rrrr>r r_rrHrrcrkrmrrrprbr8rtrupropertyrvrFrwr,rrMr-rUr"r?rErrr~rr __classcell__r1r1rr2r#sD!R    + 2  $  - >   %V#  (   6A/ /4PC 5:" 2       )r4r3rrzcCs"||j|<ttj|||dSr)r]rr:r; __setattr__)r4r3rr1r1r2r s r)r4r3rrzcCs&|j|dttj|||dSr)r]poprr:r;r)r4r3rr1r1r2r srr4r3Ztensor_or_paramcCs$t||rt||t|||dSr)r2rIsetattrrr1r1r2r s  rrcCsdd|DS)NcSs&g|]}t|tjr|nt|qSr1)rLr:rk)r}tr1r1r2r sz&_convert_to_params..r1)rr1r1r2r sr)param_or_tensorrzcCst|tjr|S|Sr)rLr:rkdetach)rr1r1r2r s rrcCsd}t|}||}|S)N)_get_dtype_size)rZ ALIGNMENTZunsharded_dtype_sizerr1r1r2r srcCstjd|dS)Nr1r)rHrZ element_sizerr1r1r2r srZ padding_numelrrorcCstj|f|||dtS)N)rror)rHZones_FLAT_PARAM_PADDING_VALUErr1r1r2r srrwarningcCs||dSr)rrr1r1r2r/ sr)\r functoolsloggingrrCenumrr itertoolsrrtypingrrrr r r r r rrrrrrrHZtorch.distributed distributedrZtorch.nnr:Ztorch.nn.functionalZ functionalr rZ$torch.distributed.fsdp._common_utilsrrrrrrZtorch.distributed.utilsrrrZtorch.nn.parameterrZ_fsdp_extensionsr r!__all__ getLoggerr)rrrrrr'r,r/Z'RESHARD_AFTER_FORWARD_HANDLE_STRATEGIESr-r0rVr%r&r?r$rJrkr"r#r;r8rrrrrrr lru_cacherrFrErrLoggerrr1r1r1r2s@        nQ