U 0-ewZ@sdZddlZddlmZmZddlmZmZddlZddl m Z ddl m Z dd l mZmZdd lmZmZed d rddlmmZerdd lmZddZddZddZddZddZed dddZdTddZddZ dd Z!d!d"Z"d#d$Z#d%d&Z$d'd(Z%d)d*Z&Gd+d,d,e'Z(d-d.Z)d/d0Z*e)d1d2Z+ed3d4d5Z,ed3d6d7Z-dUd8d9Z.dVd;d<Z/e)dWe0d=d>d?Z1dXe0d=d@dAZ2dYdBdCZ3dZdDdEZ4e*d[dFdGZ5e)d\dJdKZ6dLdMZ7GdNdOdOZ8dPdQZ9dRdSZ:dS)]zB A set of basic tensor ops compatible with tpu, gpu, and multigpu N)update_wrapperwraps)AnyMapping) PartialState)!TORCH_DISTRIBUTED_OPERATION_TYPES)DistributedTypeTensorInformation)is_torch_distributed_availableis_tpu_availableF)Z check_device)ReduceOpcCs t|tjSN) isinstancetorchTensortensorr\/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/accelerate/utils/operations.pyis_torch_tensor'src Cs2t|tjjtjjtjjtjjtjjtjjtjj Sr) rrZxpuZ FloatTensorZ ByteTensorZ IntTensorZ LongTensorZ HalfTensorZ DoubleTensorZBFloat16Tensorrrrris_torch_xpu_tensor+srcCs t|tSr)rr Z tensor_inforrris_tensor_information8srcCsVt|}|j}t|dks&|dtkr*dSt|dd}t|tsDdStdd|DS)z Checks if `x` is a `namedtuple` or not. Can have false positives, but only if a user is trying to mimic a `namedtuple` perfectly. rrF_fieldsNcss|]}t|tVqdSr)rstr).0memberrrr Hsz is_namedtuple..)type __bases__lentuplegetattrrall)dataZ data_typebasesfieldsrrr is_namedtuple<s  r)cCs(t|rt|t|St||SdS)zO Cast a generator to the same type as obj (list, tuple, or namedtuple) N)r)r list)obj generatorrrr honor_typeKsr- test_typeerror_on_other_typecst|ttfr.t|fdd|DSt|tr^t|fdd|DS|rv|fSrtdt|djdjd|S) a9 Recursively apply a function on a data structure that is a nested list/tuple/dictionary of a given base type. Args: func (`callable`): The function to recursively apply. data (nested list/tuple/dictionary of `main_type`): The data on which to apply `func` *args: Positional arguments that will be passed to `func` when applied on the unpacked data. main_type (`type`, *optional*, defaults to `torch.Tensor`): The base type of the objects to which apply `func`. error_on_other_type (`bool`, *optional*, defaults to `False`): Whether to return an error or not if after unpacking `data`, we get on an object that is not of type `main_type`. If `False`, the function will leave objects of types different than `main_type` unchanged. **kwargs: Keyword arguments that will be passed to `func` when applied on the unpacked data. Returns: The same data structure as `data` with `func` applied to every object of type `main_type`. c3s*|]"}t|fdVqdS)r.Nrecursively_apply)roargsr0funckwargsr/rrrosz$recursively_apply..cs.i|]&\}}|t|fdqS)r.r1rkvr4rr xsz%recursively_apply..zUnsupported types (z ) passed to `z?`. Only nested list/tuple/dicts of objects that are valid for `z` should be passed.) rr#r*r-rr items TypeError__name__)r6r&r/r0r5r7rr4rr2Vs( r2cst|ttfr*t|fdd|DSt|trtttrFgn dkrRgt|fdd|DSt|drz|j dWSt k r| YSXn|SdS)a Recursively sends the elements in a nested list/tuple/dictionary of tensors to a given device. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to send to a given device. device (`torch.device`): The device to send the data to. Returns: The same data structure as `tensor` with all tensors sent to the proper device. c3s|]}t|dVqdS) non_blocking skip_keysNsend_to_device)rtdevicer@rArrrsz!send_to_device..Nc s.i|]&\}}||kr|nt|dqS)r?rB)rr9rDrErrr;sz"send_to_device..to)r@) rr#r*r-rrr r<hasattrrGr=)rrFr@rArrErrCs*    rCcCsdd}t||S)aK Recursively gathers the information needed to rebuild a nested list/tuple/dictionary of tensors. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data to send to analyze. Returns: The same data structure as `data` with [`~utils.TensorInformation`] instead of tensors. cSst|j|jdS)N)shapedtype)r rIrJrrrr_get_data_structuresz/get_data_structure.._get_data_structurer1)r&rKrrrget_data_structures rLcCsdd}t||S)a: Recursively gathers the shape of a nested list/tuple/dictionary of tensors as a list. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data to send to analyze. Returns: The same data structure as `data` with lists of tensor shapes instead of tensors. cSs t|jSr)r*rIrrrr _get_shapeszget_shape.._get_shaper1)r&rMrrr get_shapes rNcCsdd}t||tdS)z Recursively initializes tensors from a nested list/tuple/dictionary of [`~utils.TensorInformation`]. Returns: The same data structure as `data` with tensors instead of [`~utils.TensorInformation`]. cSstj|jd|jiSNrJ)remptyrIrJrrrr_initialize_tensorsz.initialize_tensors.._initialize_tensorr/)r2r)Zdata_structurerQrrrinitialize_tensorssrScCslt|ttfrt|dSt|trB|D]}t||Sn t|tjsbtdt |d|j dS)a Recursively finds the batch size in a nested list/tuple/dictionary of lists of tensors. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data from which to find the batch size. Returns: `int`: The batch size. rz0Can only find the batch size of tensors but got .) rr#r*find_batch_sizerkeysrrr=r rI)r&r9rrrrUs     rUcCsdd}t||S)aS Recursively finds tensors in a nested list/tuple/dictionary and converts them to a list of numbers. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data from which to convert to regular numbers. Returns: The same data structure as `data` with lists of numbers instead of `torch.Tensor`. cSs,|}|jtjkr$|tj}|Sr)detachcpurJrbfloat16rGZfloat32tolistrrrr_convert_to_lists   z!listify.._convert_to_listr1)r&r[rrrlistifys  r\cCs"dd}t||dd}t|S)NcSs0|jdkr|d}|s&|}t|S)Nr)ndimclone is_contiguous contiguousxm all_gatherrrrr_tpu_gather_one s   z$_tpu_gather.._tpu_gather_oneTr0)r2raZ mark_step)rrcresrrr _tpu_gather s rfcCsdd}t||ddS)Ncs^jdkrds&fddttjD}tj|tj |ddS)Nrcsg|]}tqSr)rZ empty_liker_rrr "sz8_gpu_gather.._gpu_gather_one..dim) r]r^r_r`ranger distributedZget_world_sizerbcat)rZoutput_tensorsrrr_gpu_gather_ones  z$_gpu_gather.._gpu_gather_oneTrdr1)rrorrr _gpu_gathers rpc@seZdZdZdS)DistributedOperationExceptionz An exception class for distributed operations. Raised if the operation cannot be performed due to the shape of the tensors. N)r> __module__ __qualname____doc__rrrrrq)srqcstfdd}|S)zv Verifies that `tensor` is the same shape across all processes. Only ran if `PartialState().debug` is `True`. cstjtjkstjs ||Sjdj}d|krD|d}n|d}t|}t|g}|ddk r| |dt |k}|sd ddt |D}t d|d|||S) NrTrrz - cSs g|]\}}d|d|qS)zProcess z: r)rirIrrrriEsz5verify_operation..wrapper..znCannot apply desired operation due to shape mismatches. All shapes across devices must be valid. Operation: `z` Input shapes: - )rdistributed_typer NOdebugrrr>rN gather_objectcountr"join enumeraterq)r5r7 operationrZshapesoutputZare_sameZprocess_shape_strfunctionrrwrapper7s     z!verify_operation..wrapperrrrrrrverify_operation2srcstfdd}|S)z Checks that `verify_operation` failed and if so reports a more helpful error chaining the existing `DistributedOperationException`. c sXz ||WStk rR}z(jdj}td|d|W5d}~XYnXdS)NrTzError found while calling `z1`. Please see the earlier error for more details.)rqrrr>)r5r7er}rrrrVs  z"chained_operation..wrapperrrrrrchained_operationPs rcCs2tjtjkrt|Stjtkr*t|S|SdS)a4 Recursively gather tensor in a nested list/tuple/dictionary of tensors from all devices. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to gather. Returns: The same data structure as `tensor` with all tensors sent to the proper device. N)rrvr TPUrfr rprrrrgathercs  robjectcCs2ddttjD}tj||dd|DS)NcSsg|]}dqSrrrgrrrrixsz&_gpu_gather_object..cSsg|]}|D]}|q qSrr)ryxrrrri{s)rlr num_processesrrmZall_gather_object)rZoutput_objectsrrr_gpu_gather_objectwsrcCs4tjtjkrtdntjtkr,t|S|SdS)a5 Recursively gather object in a nested list/tuple/dictionary of objects from all devices. Args: object (nested list/tuple/dictionary of picklable object): The data to gather. Returns: The same data structure as `object` with all the objects sent to every device. z&gather objects in TPU is not supportedN)rrvr rNotImplementedErrorr rrrrrry~s   rycCsddd}t||d|dS)NrcSstjj||d|S)Nsrc)rrm broadcast)rrrrr_gpu_broadcast_onesz*_gpu_broadcast.._gpu_broadcast_oneT)r0r)rr1)r&rrrrr_gpu_broadcasts rbroadcast tensorcsht|ttfr*t|fddt|DSt|trRt|fdd|DSt |fddS)Nc3s(|] \}}t|d|dVqdS)rhnameN_tpu_broadcast)rrurDrrrrsz!_tpu_broadcast..cs(i|] \}}|t|d|dqS)rhrrr8rrrr;sz"_tpu_broadcast..cs|Srrrrrrz _tpu_broadcast..) rr*r#r-r|rr r<ra mesh_reduce)rrrr)rrrrs  r from_processcCs<tjtjkrt||ddStjtkr4t||dS|SdS)a Recursively broadcast tensor in a nested list/tuple/dictionary of tensors to all devices. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to gather. from_process (`int`, *optional*, defaults to 0): The process from which to send the data Returns: The same data structure as `tensor` with all tensors broadcasted to the proper device. zaccelerate.utils.broadcast)rrrN)rrvr rrr r)rrrrrrs   rcs\tjtjkr.r) rrvr rr|rarr rrmbroadcast_object_list)Z object_listrrur+rrrrs  rcCsdd}t|||S)aN Recursively takes a slice in a nested list/tuple/dictionary of tensors. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data to slice. tensor_slice (`slice`): The slice to take. Returns: The same data structure as `data` with all the tensors slices. cSs||Srr)r tensor_slicerrr _slice_tensorsz$slice_tensors.._slice_tensorr1)r&rZ process_indexrrrrr slice_tensorssrcstdttfr..rjN concatenaterr&rkrrrszconcatenate..cs(i|] tfddDdqS)csg|] }|qSrrrr9rrrisz*concatenate...rjrrrrrr;szconcatenate..z%Can only concatenate tensors but got rj) rr#r*r-rlr"rr rVrrr=rnrrrrrs *(rcCsddd}t||d|||dS)a3 Recursively pad the tensors in a nested list/tuple/dictionary of tensors from all devices to the same size so they can safely be gathered. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to gather. dim (`int`, *optional*, defaults to 0): The dimension on which to pad. pad_index (`int`, *optional*, defaults to 0): The value with which to pad. pad_first (`bool`, *optional*, defaults to `False`): Whether to pad at the beginning or the end. rFc st|jkr|Stj|j|jdd}t|}tfdd|D|jkr\|S|jt}|<| t ||}|rt fddt t|D}n t fddt t|D}|||<|S)N)rFc3s|]}|VqdSrr)rsrjrrrszFpad_across_processes.._pad_across_processes..c3s0|](}|kr tntdVqdSrslicerrurkmax_sizeold_sizerrrsc3s,|]$}|krtdntdVqdS)rNrr)rkrrrrs) r"rIrrrFrrXmaxr*Z new_zerosr#rl) rrk pad_index pad_firstsizesizesnew_sizeZ new_tensorindicesrrr_pad_across_processess$   z3pad_across_processes.._pad_across_processesT)r0rkrr)rrFr1)rrkrrrrrrpad_across_processess rmean?cCsddd}t||d||dS)aX Recursively reduce the tensors in a nested list/tuple/dictionary of lists of tensors across all processes by the mean of a given operation. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to reduce. reduction (`str`, *optional*, defaults to `"mean"`): A reduction method. Can be of "mean", "sum", or "none" scale (`float`, *optional*): A default scaling value to be applied after the reduce, only valied on XLA. Returns: The same data structure as `data` with all the tensors reduced. rrcSslt}|}|jtjkr|S|jtjkr:td||n|jjt krVt j |t j |dkrh||j}|S)Nsumr)rr^rvr rwrraZ all_reducevaluer rrmrZSUMr)r reductionscalestateZ cloned_tensorrrr_reduce_across_processes7s    z(reduce.._reduce_across_processesT)r0rr)rrr1)rrrrrrrreduce%s rcCsdd}dd}t|||dS)av Recursively converts the elements nested list/tuple/dictionary of tensors in FP16/BF16 precision to FP32. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to convert from FP16/BF16 to FP32. Returns: The same data structure as `tensor` with all tensors that were in FP16/BF16 precision converted to FP32. cSs|Sr)floatrrrr_convert_to_fp32Usz)convert_to_fp32.._convert_to_fp32cSst|do|jtjtjfkSrO)rHrJrZfloat16rYrrrr_is_fp16_bf16_tensorXsz-convert_to_fp32.._is_fp16_bf16_tensorrRr1)rrrrrrconvert_to_fp32Is rc@s(eZdZdZddZddZddZdS) ConvertOutputsToFp32ad Decorator to apply to a function outputing tensors (like a model forward pass) that ensures the outputs in FP16 precision will be convert back to FP32. Args: model_forward (`Callable`): The function which outputs we want to treat. Returns: The same function as `model_forward` but with converted outputs. cCs||_t||dSr) model_forwardr)selfrrrr__init__kszConvertOutputsToFp32.__init__cOst|j||Sr)rr)rr5r7rrr__call__oszConvertOutputsToFp32.__call__cCstddS)NzCannot pickle a prepared model with automatic mixed precision, please unwrap the model with `Accelerator.unwrap_model(model)` before pickling it.)pickle PicklingError)rrrr __getstate__rsz!ConvertOutputsToFp32.__getstate__N)r>rrrsrtrrrrrrrr^s rcstfdd}|_|S)Ncs ||Srr)r5r7rrrforward{sz(convert_outputs_to_fp32..forward)r __wrapped__)rrrrrconvert_outputs_to_fp32xs rcCszt|tr2|D]}t|}|dk r|SqnDt|ttfrd|D]}t|}|dk rD|SqDnt|tjrv|jSdS)z Finds the device on which a nested dict/list/tuple of tensors lies (assuming they are all on the same device). Args: (nested list/tuple/dictionary of `torch.Tensor`): The data we want to know the device of. N) rrvalues find_devicer#r*rrrF)r&r+rFrrrrs     r)FN)r)rr)r)r)NN)r)rrF)rr);rtr functoolsrrtypingrrrrr constantsr dataclassesr r Zimportsr r Ztorch_xla.core.xla_modelcoreZ xla_modelraZtorch.distributedrrrrr)r-r2rCrLrNrSrUr\rfrp Exceptionrqrrrrryrrintrrrrrrrrrrrrrrs`      3 %       . #