U Ç9%ezpã@sddlmZmZmZmZddlZddlmZddlmZm Z m Z m Z m Z m Z mZmZmZmZmZmZddlmZddgZGd d„deƒZd d e›d e›d e›d e›d e›d e_deeeeeeeeeeeeeeeeeeeeeeeeeeeefeeedœdd„Zeeeeeeeeeeeeeeeeeeeeeefeeeeedœdd„Zeeeeeeeeeeeeeeeeeeeeeefeeeeedœdd„Zeeeeeeeeeeeeeeeeeeeeeefeeeeeddœdd„ZdS)é)ÚListÚOptionalÚUnionÚTupleN)ÚTensoré) Ú OptimizerÚparams_tÚ_use_grad_for_differentiableÚ _get_valueÚ_stack_if_compilingÚ_dispatch_sqrtÚ_default_to_fused_or_foreachÚ_capturable_docÚ_differentiable_docÚ _foreach_docÚ _fused_docÚ _maximize_doc)Ú$_get_fused_kernels_supported_devicesÚAdamÚadamcs€eZdZdddddddœeeeefeeefeeee eeeee edœ ‡fd d „Z ‡fd d „Z d d„Z e ddd„ƒZ‡ZS)rçü©ñÒMbP?©gÍÌÌÌÌÌì?g+‡ÙÎ÷ï?ç:Œ0âŽyE>rFN)ÚforeachÚmaximizeÚ capturableÚdifferentiableÚfused) ÚparamsÚlrÚbetasÚepsÚ weight_decayÚamsgradrrrrrc s8d|kstd|›ƒ‚t|tƒr0|r0| s0tdƒ‚d|ksFtd|›ƒ‚d|dkr^dksrntd|d›ƒ‚d|dkrŠdksžntd |d›ƒ‚d|ks´td |›ƒ‚t|||||||| | | d } tƒ || ¡| r4| rðtd ƒ‚d |_tƒ‰t ‡fdd„|j Dƒƒs&tdˆ›dƒ‚|r4tdƒ‚dS)NçzInvalid learning rate: úElr as a Tensor is not supported for capturable=False and foreach=TruezInvalid epsilon value: rgð?z#Invalid beta parameter at index 0: rz#Invalid beta parameter at index 1: zInvalid weight_decay value: ) r r!r"r#r$rrrrrz)`fused` does not support `differentiable`Tc3s2|]*}|dD]}|jjˆko&t |¡VqqdS)rN)ÚdeviceÚtypeÚtorchZis_floating_point)Ú.0ZpgÚp©Zfused_supported_devices©úO/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/torch/optim/adam.pyÚ 8s  ÿ z Adam.__init__..zX`fused=True` requires all the params to be floating point Tensors of supported devices: Ú.z0`fused` and `foreach` cannot be `True` together.) Ú ValueErrorÚ isinstancerÚdictÚsuperÚ__init__Ú RuntimeErrorZ_step_supports_amp_scalingrÚallÚ param_groups) Úselfrr r!r"r#r$rrrrrÚdefaults©Ú __class__r,r.r5sB ý þ z Adam.__init__cs´tƒ |¡|jD]L}| dd¡| dd¡| dd¡| dd¡| dd¡| dd¡qt|j ¡ƒ}t|ƒdkoŠt  |dd ¡}|s°|D]}t  t |d ƒ¡|d <q”dS) Nr$FrrrrrrÚstep) r4Ú __setstate__r8Ú setdefaultÚlistÚstateÚvaluesÚlenr)Ú is_tensorÚtensorÚfloat)r9rAÚgroupZ state_valuesZstep_is_tensorÚsr;r-r.r>As       zAdam.__setstate__c CsZ|dD]J}|jdk r| |¡|jjr2tdƒ‚| |j¡|j|} t| ƒdkrÊ|dsd|drxtjdtj|j dnt  d¡| d <tj |tj d | d <tj |tj d | d <|d rÊtj |tj d | d<| | d ¡| | d ¡|d rü| | d¡|dr| d j rtdƒ‚|drFt |d¡rF|dsFtdƒ‚| | d ¡qdS)NrzJAdam does not support sparse gradients, please consider SparseAdam insteadrrrr-)Zdtyper'r%r=)Z memory_formatÚexp_avgÚ exp_avg_sqr$Úmax_exp_avg_sqrzB`requires_grad` is not supported for `step` in differentiable moderr r&)ÚgradÚappendZ is_sparser6rArCr)ZzerosrFr'rEZ zeros_likeZpreserve_formatZ requires_gradrD) r9rGÚparams_with_gradÚgradsÚexp_avgsÚ exp_avg_sqsÚmax_exp_avg_sqsÚ state_stepsr+rAr-r-r.Ú _init_groupPs6      ÿÿý$zAdam._init_groupc CsÚ| ¡d}|dk r.t ¡ |ƒ}W5QRX|jD] }g}g}g}g}g}g} |d\} } | ||||||| ¡t|||||| |d| | |d|d|d|d|d|d |d |d t|d dƒt|d dƒdq4|S)z±Performs a single optimization step. Args: closure (Callable, optional): A closure that reevaluates the model and returns the loss. Nr!r$r r#r"rrrrrÚ grad_scaleÚ found_inf) r$Úbeta1Úbeta2r r#r"rrrrrrUrV)Z _cuda_graph_capture_health_checkr)Z enable_gradr8rTrÚgetattr) r9ÚclosureZlossrGrNrOrPrQrRrSrWrXr-r-r.r=‚sX   ù   íz Adam.step)rrrrF)N)Ú__name__Ú __module__Ú __qualname__r rrFrrÚboolrr5r>rTr r=Ú __classcell__r-r-r;r.rs8úô  ô2 2a Implements Adam algorithm. .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \gamma \text{ (lr)}, \beta_1, \beta_2 \text{ (betas)},\theta_0 \text{ (params)},f(\theta) \text{ (objective)} \\ &\hspace{13mm} \lambda \text{ (weight decay)}, \: \textit{amsgrad}, \:\textit{maximize} \\ &\textbf{initialize} : m_0 \leftarrow 0 \text{ ( first moment)}, v_0\leftarrow 0 \text{ (second moment)},\: \widehat{v_0}^{max}\leftarrow 0\\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}\textbf{if} \: \textit{maximize}: \\ &\hspace{10mm}g_t \leftarrow -\nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}\textbf{else} \\ &\hspace{10mm}g_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}\textbf{if} \: \lambda \neq 0 \\ &\hspace{10mm} g_t \leftarrow g_t + \lambda \theta_{t-1} \\ &\hspace{5mm}m_t \leftarrow \beta_1 m_{t-1} + (1 - \beta_1) g_t \\ &\hspace{5mm}v_t \leftarrow \beta_2 v_{t-1} + (1-\beta_2) g^2_t \\ &\hspace{5mm}\widehat{m_t} \leftarrow m_t/\big(1-\beta_1^t \big) \\ &\hspace{5mm}\widehat{v_t} \leftarrow v_t/\big(1-\beta_2^t \big) \\ &\hspace{5mm}\textbf{if} \: amsgrad \\ &\hspace{10mm}\widehat{v_t}^{max} \leftarrow \mathrm{max}(\widehat{v_t}^{max}, \widehat{v_t}) \\ &\hspace{10mm}\theta_t \leftarrow \theta_{t-1} - \gamma \widehat{m_t}/ \big(\sqrt{\widehat{v_t}^{max}} + \epsilon \big) \\ &\hspace{5mm}\textbf{else} \\ &\hspace{10mm}\theta_t \leftarrow \theta_{t-1} - \gamma \widehat{m_t}/ \big(\sqrt{\widehat{v_t}} + \epsilon \big) \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} For further details regarding the algorithm we refer to `Adam: A Method for Stochastic Optimization`_. a Args: params (iterable): iterable of parameters to optimize or dicts defining parameter groups lr (float, Tensor, optional): learning rate (default: 1e-3). A tensor LR is not yet supported for all our implementations. Please use a float LR if you are not also specifying fused=True or capturable=True. betas (Tuple[float, float], optional): coefficients used for computing running averages of gradient and its square (default: (0.9, 0.999)) eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-8) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) amsgrad (bool, optional): whether to use the AMSGrad variant of this algorithm from the paper `On the Convergence of Adam and Beyond`_ (default: False) z zÄ .. _Adam\: A Method for Stochastic Optimization: https://arxiv.org/abs/1412.6980 .. _On the Convergence of Adam and Beyond: https://openreview.net/forum?id=ryQu7f-RZ F)rrOrPrQrRrSrrrrrUrVr$rWrXr r#r"rc Csú| dkr8|dkr8t||dd\}}|r8t|tƒr8|s8d}| dkrDd} |dkrPd}tj ¡sttdd„|Dƒƒsttdƒ‚|rŠtj  ¡rŠtdƒ‚| r tj  ¡r tdƒ‚| r´tj  ¡s´t }n|rÈtj  ¡sÈt }nt }|||||||| | |||||||| | d dS) zmFunctional API that performs Adam algorithm computation. See :class:`~torch.optim.Adam` for details. NF)Z use_fusedcss|]}t|tjƒVqdS©N)r2r)r)r*Útr-r-r.r/(szadam..zPAPI has changed, `state_steps` argument must contain a list of singleton tensorsz6torch.jit.script not supported with foreach optimizersz4torch.jit.script not supported with fused optimizers) r$rWrXr r#r"rrrrUrV) rr2rr)Ú_utilsÚ is_compilingr7r6ÚjitÚ is_scriptingÚ _fused_adamÚ_multi_tensor_adamÚ_single_tensor_adam)rrOrPrQrRrSrrrrrUrVr$rWrXr r#r"rÚ_Úfuncr-r-r.rþsJð)rrOrPrQrRrSrUrVr$rWrXr r#r"rrrc Cs¤|dkr|dkst‚tj ¡r,t| tƒs,t‚t|ƒD]h\}}|sJ||n|| }||}||}||}tj ¡sš|rš|j r†|j sš|j r’|j sštdƒ‚|d7}| dkr¸|j || d}t  |¡rt  |¡}t  |¡}t  |¡}|røt  ||¡||<t  |¡}| |d| ¡| | ¡j|| ¡d| d|s<|rð|}d| |}d| |}| |}| ¡}| ¡}|rÈ|rŠ|| ¡}n||}|| t ||¡¡|| ¡|| | |¡}n| ¡|| | |¡}| ||¡nŠt|ƒ}d| |}d| |}| |}t|ƒ}|rVtj|||||d|| ¡| | ¡}n| ¡| | ¡}|j||| d|r4t  ||¡r4t ||¡||<q4dS)NzGIf capturable=True, params and state_steps must be CUDA or XLA tensors.rr©Úalpha)Úvalue)Úout)ÚAssertionErrorr)rdrer2rFÚ enumeraterbrcÚis_cudaZis_xlaÚaddÚ is_complexÚ view_as_realZlerp_Zmul_Zaddcmul_ZconjÚnegÚsqrtÚcloneZcopy_ÚmaximumZadd_Zaddcdiv_r r Zview_as_complex)rrOrPrQrRrSrUrVr$rWrXr r#r"rrrÚiÚparamrLrIrJZstep_tr=Úbias_correction1Úbias_correction2Ú step_sizeZ step_size_negÚbias_correction2_sqrtrKÚdenomr-r-r.rhJst ÿÿÿÿþ           rhc sÐt|ƒdkrdStˆtƒr&|s&tdƒ‚tj ¡sT|rTtdd„t||ƒDƒƒsTt dƒ‚|dkrd|dksht ‚|rtt dƒ‚t   ||||||g¡}|  ¡D]6\\}}}}}}}|r¶t  |¡}dd„|Dƒ}d d„|Dƒ}d d„|Dƒ}d d„|Dƒ}d d„|Dƒ}t |d ¡| dkr:|r*tj||| dntj||| d}t ||d ˆ¡t |ˆ¡t |||d ˆ¡~|r2t ˆ|¡}t ˆ|¡}t |d ¡t |d ¡t |¡t |ˆ¡t |¡t |¡|}|}|rôt ||¡t |¡}n t |¡}t ||¡t || ¡t ||¡t |||¡q’‡fdd„|Dƒ}‡fdd„|Dƒ}t‡fdd„|Dƒƒ}dd„|Dƒ}|r˜t ||¡t |¡}n t |¡}t ||¡t || ¡t ||||¡q’dS)Nrr&css|]\}}|jo|jVqdSr`)rq)r*r+r=r-r-r.r/Ósz%_multi_tensor_adam..z@If capturable=True, params and state_steps must be CUDA tensors.z#_foreach ops don't support autogradcSs$g|]}t |¡rt |¡n|‘qSr-©r)rsrt©r*Úxr-r-r.Ú ész&_multi_tensor_adam..cSs$g|]}t |¡rt |¡n|‘qSr-r€rr-r-r.rƒêscSs$g|]}t |¡rt |¡n|‘qSr-r€rr-r-r.rƒëscSs$g|]}t |¡rt |¡n|‘qSr-r€rr-r-r.rƒìscSs$g|]}t |¡rt |¡n|‘qSr-r€rr-r-r.rƒísrrkcsg|]}dˆt|ƒ‘qS©r©r ©r*r=)rWr-r.rƒ'scsg|]}dˆt|ƒ‘qSr„r…r†)rXr-r.rƒ(scsg|]}ˆ|d‘qS)éÿÿÿÿr-©r*Úbc)r r-r.rƒ*scSsg|] }t|ƒ‘qSr-)r rˆr-r-r.rƒ,s)rCr2rr6r)rbrcr7ÚziprorÚ"_group_tensors_by_device_and_dtyperBZ _foreach_negÚ _foreach_add_Z _foreach_addZ_foreach_lerp_Z _foreach_mul_Z_foreach_addcmul_Z _foreach_powÚ _foreach_sub_Z _foreach_neg_Z _foreach_div_Z_foreach_reciprocal_Z_foreach_sqrt_Z_foreach_maximum_Z _foreach_sqrtZ_foreach_addcdiv_r )rrOrPrQrRrSrUrVr$rWrXr r#r"rrrÚgrouped_tensorsÚ device_paramsÚ device_gradsÚdevice_exp_avgsÚdevice_exp_avg_sqsÚdevice_max_exp_avg_sqsÚdevice_state_stepsrir{r|r}r~Zexp_avg_sq_sqrtr-)rWrXr r.rg¹sˆ ÿ ÿ ù                        rg)rrOrPrQrRrSrUrVr$rWrXr r#r"rrrÚreturnc Cs‚|sdS|rtdƒ‚|dk r&|j|ind}|dk r<|j|ind}t| tƒrbt| jƒdkrb| j| ind}t ||||||g¡}| ¡D]ø\\}}\\}}}}}}}d\}}|dk rÒ||krÊ|j|dd||<||}|dk rü||krô|j|dd||<||}|dk r*||kr*| j|dd||<||} t   |d¡t j |||||||| | | | | |||d|dk r„t   ||gt |ƒ¡q„dS) Nz9Adam with fused=True does not support differentiable=TrueÚcpu)NNT)Ú non_blocking)r'r—r) r$r rWrXr#r"rrUrV)r6r'r2rÚstrrr‹ÚitemsÚtor)rŒZ _fused_adam_rrC)rrOrPrQrRrSrUrVr$rWrXr r#r"rrrZgrad_scale_dictZfound_inf_dictZlr_dictrŽr'rirrr‘r’r“r”Zdevice_grad_scaleZdevice_found_infr-r-r.rf<sd&ÿû ñrf)NFFNNN)Útypingrrrrr)rZ optimizerrr r r r r rrrrrrZtorch.utils._foreach_utilsrÚ__all__rÚ__doc__r^rFrrhrgrfr-r-r-r.ÚsÌ 8 /&ñðïîíÚJó ë L ï o ï  í