U 9%eO"@sddlZddlmZddlZddlmZddlmZmZddlm Z e e Z Gdddej ZGd d d ej ZGd d d ej ZGd ddej ZGdddej ZGdddej ZGdddej ZGdddej ZGdddej ZGdddej ZGdddej ZGdddej ZGdd d eZeed!d"d#feeed$d%ifeeeeeeejeejejeeejd&ZeeZ d'd(Z!e!d)Z"e!d*Z#e!d+Z$e!d,Z%e!d-Z&e!d.Z'e!d/Z(e!d0Z)dS)1N) OrderedDict)version)Tensornn)loggingcs0eZdZdZfddZeedddZZS)PytorchGELUTanha A fast C implementation of the tanh approximation of the GeLU activation function. See https://arxiv.org/abs/1606.08415. This implementation is equivalent to NewGELU and FastGELU but much faster. However, it is not an exact numerical match due to rounding errors. cs6tttjtdkr2tdtjddS)Nz1.12.0zYou are using torch==zM, but torch>=1.12.0 is required to use PytorchGELUTanh. Please upgrade torch.)super__init__rparsetorch __version__ ImportErrorself __class__W/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/transformers/activations.pyr %s   zPytorchGELUTanh.__init__inputreturncCstjj|ddS)Ntanh)Z approximate)r functionalgelurrrrrforward-szPytorchGELUTanh.forward__name__ __module__ __qualname____doc__r rr __classcell__rrrrrs rc@s eZdZdZeedddZdS)NewGELUActivationz Implementation of the GELU activation function currently in Google BERT repo (identical to OpenAI GPT). Also see the Gaussian Error Linear Units paper: https://arxiv.org/abs/1606.08415 rc Cs6d|dttdtj|dt|dS)N??@Hm?g@)r rmathsqrtpipowrrrrr7szNewGELUActivation.forwardNrrr r!rrrrrrr#1sr#csHeZdZdZd edfdd ZeedddZeedd d ZZ S) GELUActivationa Original Implementation of the GELU activation function in Google BERT repo when initially created. For information: OpenAI GPT's GELU is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) This is now written in C in nn.functional Also see the Gaussian Error Linear Units paper: https://arxiv.org/abs/1606.08415 F)use_gelu_pythoncs&t|r|j|_n tjj|_dSN)r r _gelu_pythonactrrr)rr.rrrr Cs  zGELUActivation.__init__rcCs |ddt|tdS)Nr$r%r&)r erfr(r)rrrrr0JszGELUActivation._gelu_pythoncCs ||Sr/r1rrrrrMszGELUActivation.forward)F) rrr r!boolr rr0rr"rrrrr-;sr-c@s eZdZdZeedddZdS)FastGELUActivationz} Applies GELU approximation that is slower than QuickGELU but more accurate. See: https://github.com/hendrycks/GELUs rcCs*d|dt|ddd||S)Nr$r%g3E?r')r rrrrrrVszFastGELUActivation.forwardNr,rrrrr5Qsr5c@s eZdZdZeedddZdS)QuickGELUActivationzr Applies GELU approximation that is fast but somewhat inaccurate. See: https://github.com/hendrycks/GELUs rcCs|td|S)NgZd;?)r sigmoidrrrrr_szQuickGELUActivation.forwardNr,rrrrr6Zsr6cs8eZdZdZeedfdd ZeedddZZS)ClippedGELUActivationa Clip the range of possible GeLU outputs between [min, max]. This is especially useful for quantization purpose, as it allows mapping negatives values in the GeLU spectrum. For more information on this trick, please refer to https://arxiv.org/abs/2004.09602. Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))). See https://arxiv.org/abs/1606.08415 minmaxcs8||krtd|d|dt||_||_dS)Nzmin should be < max (got min: z, max: )) ValueErrorr r r:r;)rr:r;rrrr ps  zClippedGELUActivation.__init__)xrcCstt||j|jSr/)r Zcliprr:r;)rr>rrrrxszClippedGELUActivation.forward) rrr r!floatr rrr"rrrrr8cs r8cs0eZdZdZfddZeedddZZS)AccurateGELUActivationz Applies GELU approximation that is faster than default and more accurate than QuickGELU. See: https://github.com/hendrycks/GELUs Implemented along with MEGA (Moving Average Equipped Gated Attention) cs ttdtj|_dS)N)r r r(r)r*precomputed_constantrrrrr s zAccurateGELUActivation.__init__rc Cs,d|dt|j|dt|dS)Nr$rr')r rrBr+rrrrrszAccurateGELUActivation.forwardrrrrrr@|s r@c@s eZdZdZeedddZdS)SiLUActivationa See Gaussian Error Linear Units (Hendrycks et al., https://arxiv.org/abs/1606.08415) where the SiLU (Sigmoid Linear Unit) was originally introduced and coined, and see Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning (Elfwing et al., https://arxiv.org/abs/1702.03118) and Swish: a Self-Gated Activation Function (Ramachandran et al., https://arxiv.org/abs/1710.05941v1) where the SiLU was experimented with later. rcCs tj|Sr/)rrsilurrrrrszSiLUActivation.forwardNr,rrrrrDsrDcs@eZdZdZfddZeedddZeedddZZS) MishActivationz See Mish: A Self-Regularized Non-Monotonic Activation Function (Misra., https://arxiv.org/abs/1908.08681). Also visit the official repository for the paper: https://github.com/digantamisra98/Mish cs8tttjtdkr*|j|_n tj j |_dS)Nz1.9.0) r r rr r r _mish_pythonr1rrmishrrrrr s  zMishActivation.__init__rcCs|ttj|Sr/)r rrrZsoftplusrrrrrGszMishActivation._mish_pythoncCs ||Sr/r3rrrrrszMishActivation.forward) rrr r!r rrGrr"rrrrrFs rFc@s eZdZdZeedddZdS)LinearActivationz[ Applies the linear activation function, i.e. forwarding input directly to output. rcCs|Sr/rrrrrrszLinearActivation.forwardNr,rrrrrIsrIc@seZdZdZdddZdS)LaplaceActivationz Applies elementwise activation based on Laplace function, introduced in MEGA as an attention activation. See https://arxiv.org/abs/2209.10655 Inspired by squared relu, but with bounded range and gradient for better stability 绹۞? ^/ ?cCs*|||td}ddt|S)Nr&r$r%)divr(r)r r2)rrmusigmarrrrszLaplaceActivation.forwardN)rKrLrrr r!rrrrrrJsrJc@seZdZdZddZdS)ReLUSquaredActivationzX Applies the relu^2 activation introduced in https://arxiv.org/abs/2109.08668v2 cCstj|}t|}|Sr/)rrrelur Zsquare)rrZ relu_appliedZsquaredrrrrs  zReLUSquaredActivation.forwardNrPrrrrrQsrQcseZdZfddZZS)ClassInstantiercs0t|}t|tr|n|if\}}|f|Sr/)r __getitem__ isinstancetuple)rkeycontentclskwargsrrrrTs zClassInstantier.__getitem__)rrr rTr"rrrrrSsrSi r9r.T)rZgelu_10 gelu_fastgelu_new gelu_pythonZgelu_pytorch_tanhZ gelu_accurateZlaplacelinearrH quick_gelurRZrelu2Zrelu6r7rEZswishrcCs0|tkrt|Std|dttdS)Nz function z not found in ACT2FN mapping )ACT2FNKeyErrorlistkeys)Zactivation_stringrrrget_activationsrer^r]rr\r`rErHr_)*r( collectionsrr packagingrrrutilsrZ get_loggerrloggerModulerr#r-r5r6r8r@rDrFrIrJrQrSZReLUZReLU6ZSigmoidZTanhZACT2CLSrarer^r]rr\r`rErHZ linear_actrrrrs`