U *-eg@sjUddlmZmZmZmZddlZddlmmZ ddlm Z gZ ee e d<ejjGdddZdS))DictListOptionalTupleN)Tensor__all__c @sdeZdZdeeeeeefeeeeeeed ddZee ed d d Z ee ed d dZ dS)_FunctionalAdamMbP?g?g+?:0yE>F) paramslrbetaseps weight_decayamsgradmaximizeforeachfused_allow_empty_param_listc Csd|kstd|d|ks,td|d|dkrDdksXntd|dd|dkrpdksntd|dd|kstd ||||d|d|d |_||_||_||_| |_tjt tj t t tj ffi|_ t |dkr | s td d |i|_dS) Nr zInvalid learning rate: zInvalid epsilon value: rg?z#Invalid beta parameter at index 0: z#Invalid beta parameter at index 1: zInvalid weight_decay value: )rrbeta1beta2rz%optimizer got an empty parameter listr ) ValueErrordefaultsrrrrtorchjitZannotaterrstrstatelen param_group) selfr rrrrrrrrrr#h/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/torch/distributed/optim/functional_adam.py__init__s0 $z_FunctionalAdam.__init__)paramgradc CsTg}g}g}g}g}g}|dk r4||||||jkri|j|<|j|} td| d<tj|tjd| d<tj|tjd| d<|jrtj|tjd| d<|j|} || d|| d|jr|| d|| dtXtj |||||||j|j |j d|j d |j d |j d |j d |j |j ddd W5QRXdS)zo Similar to step, but operates on a single parameter and optionally a gradient tensor. Nr stepZ memory_formatexp_avg exp_avg_sqmax_exp_avg_sqrrrrr rrrrrrrrrZ grad_scaleZ found_inf)appendrrtensor zeros_likepreserve_formatrno_gradFadamrrrr) r"r&r'params_with_gradgradsexp_avgs exp_avg_sqsmax_exp_avg_sqs state_stepsrr#r#r$ step_paramAsh          z_FunctionalAdam.step_param) gradientsc Cs|jd}g}g}g}g}g}g}t|t|krXtddt|ddt|t|jd|D]\} } | dk rh|| || | |jkri|j| <|j| } td| d<tj| tj d| d <tj| tj d| d <|j rtj| tj d| d <|j| } || d || d |j r6|| d || dqht Xt j |||||||j |j|jd |jd |jd|jd|jd|j|jdddW5QRXdS)Nr zEthe gradients passed in does not equal to the size of the parameters!zParams length: z. zGradients length: r r(r)r*r+r,rrrrrr-)r!r rzipr.rrr/r0r1rr2r3r4rrrr) r"r<r r5r6r7r8r9r:r&Zgradientrr#r#r$r({s|            z_FunctionalAdam.stepN) r r r r FFFFF) __name__ __module__ __qualname__rrfloatrboolr%rr;r(r#r#r#r$rs.  ,:r)typingrrrrrZtorch.optim._functionalZoptimZ _functionalr3rrr__annotations__rscriptrr#r#r#r$s