U ,-e^@sdZddlZddlmZmZmZddlZddlZddlmZddl m Z m Z m Z ddl mZddlmZmZmZdd lmZdd lmZmZmZmZd d lmZeeZd ZdZ d dddgZ!dZ"dZ#dgZ$dZ%dZ&ee'dddZ(d3ee'efe)dddZ*Gdddej+Z,Gdddej-Z.Gdd d ej+Z/Gd!d"d"ej+Z0Gd#d$d$ej+Z1Gd%d&d&ej+Z2Gd'd(d(ej+Z3Gd)d*d*ej+Z4Gd+d,d,eZ5ed-e%Gd.d/d/e5Z6ed0e%Gd1d2d2e5Z7dS)4z PyTorch EfficientNet model.N)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BaseModelOutputWithNoAttention(BaseModelOutputWithPoolingAndNoAttention$ImageClassifierOutputWithNoAttention)PreTrainedModel)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardlogging)EfficientNetConfigrzgoogle/efficientnet-b7iztabby, tabby cataN This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`EfficientNetConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. aB Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`AutoImageProcessor.__call__`] for details. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. )config num_channelscCsJ|j}||j9}t|t||d||}|d|krB||7}t|S)z< Round number of filters based on depth multiplier. g?)Z depth_divisorZwidth_coefficientmaxint)rrZdivisorZnew_dimrw/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/models/efficientnet/modeling_efficientnet.py round_filtersVs   rT) kernel_sizeadjustcCsrt|tr||f}|dd|ddf}|rR|dd|d|dd|dfS|d|d|d|dfSdS)aJ Utility function to get the tuple padding value for the depthwise convolution. Args: kernel_size (`int` or `tuple`): Kernel size of the convolution layers. adjust (`bool`, *optional*, defaults to `True`): Adjusts padding value to apply to right and bottom sides of the input. rrrN) isinstancer)rrZcorrectrrr correct_pades $r!cs:eZdZdZedfdd ZejejdddZZ S)EfficientNetEmbeddingszL A module that corresponds to the stem module of the original work. rcshtt|d|_tjdd|_tj|j|jddddd|_ tj |j|j |j d |_ t|j|_dS) N )rrrrpaddingr rvalidFrstrider&bias)epsmomentum)super__init__rout_dimr ZeroPad2dr&Conv2dr convolution BatchNorm2dbatch_norm_epsbatch_norm_momentum batchnormr hidden_act activationselfr __class__rrr.~s  zEfficientNetEmbeddings.__init__) pixel_valuesreturncCs,||}||}||}||}|SN)r&r2r6r8)r:r=featuresrrrforwards     zEfficientNetEmbeddings.forward) __name__ __module__ __qualname____doc__rr.torchTensorrA __classcell__rrr;rr"ys r"cseZdZdfdd ZZS) EfficientNetDepthwiseConv2drr rTzerosc s*||} tj|| |||||||d dS)N) in_channels out_channelsrr)r&dilationgroupsr* padding_mode)r-r.) r:rKZdepth_multiplierrr)r&rMr*rOrLr;rrr.s z$EfficientNetDepthwiseConv2d.__init__)rr rrrTrJ)rBrCrDr.rHrrr;rrIsrIcs@eZdZdZeeeedfdd Zejej dddZ Z S)EfficientNetExpansionLayerz_ This corresponds to the expansion phase of each block in the original implementation. rin_dimr/r)csBttj||dddd|_tj||jd|_t|j |_ dS)NrsameFrKrLrr&r*) num_featuresr+) r-r.rr1 expand_convr3r4 expand_bnr r7 expand_act)r:rrRr/r)r;rrr.s z#EfficientNetExpansionLayer.__init__ hidden_statesr>cCs"||}||}||}|Sr?)rVrWrXr:rZrrrrAs   z"EfficientNetExpansionLayer.forward) rBrCrDrErrr.rF FloatTensorrGrArHrrr;rrPs rPcsBeZdZdZeeeeedfdd Zej ej dddZ Z S)EfficientNetDepthwiseLayerzk This corresponds to the depthwise convolution phase of each block in the original implementation. rrRr)radjust_paddingcsvt||_|jdkrdnd}t||d}tj|d|_t||||dd|_tj ||j |j d|_ t |j|_dS) Nrr'rS)rr%Fr(rUr+r,)r-r.r)r!rr0depthwise_conv_padrIdepthwise_convr3r4r5depthwise_normr r7 depthwise_act)r:rrRr)rr_Zconv_padr&r;rrr.s$  z#EfficientNetDepthwiseLayer.__init__rYcCs6|jdkr||}||}||}||}|S)Nr)r)rarbrcrdr[rrrrAs      z"EfficientNetDepthwiseLayer.forward rBrCrDrErrboolr.rFr\rGrArHrrr;rr]sr]csBeZdZdZd eeeedfdd Zej ej dddZ Z S) EfficientNetSqueezeExciteLayerzl This corresponds to the Squeeze and Excitement phase of each block in the original implementation. FrrR expand_dimexpandcst|r|n||_tdt||j|_tjdd|_ tj |j|jddd|_ tj |j|jddd|_ t |j|_t|_dS)Nr)Z output_sizerS)rKrLrr&)r-r.dimrrZsqueeze_expansion_ratioZdim_serZAdaptiveAvgPool2dsqueezer1reducerjr r7 act_reduceZSigmoid act_expand)r:rrRrirjr;rrr.s$  z'EfficientNetSqueezeExciteLayer.__init__rYcCsF|}||}||}||}||}||}t||}|Sr?)rlrmrnrjrorFmul)r:rZinputsrrrrAs      z&EfficientNetSqueezeExciteLayer.forward)Frerrr;rrgsrgcsHeZdZdZeeeeeedfdd Ze j e j e j dddZ Z S)EfficientNetFinalBlockLayerz[ This corresponds to the final phase of each block in the original implementation. rrRr/r) drop_rateid_skipcsXt|dko| |_tj||dddd|_tj||j|jd|_ tj |d|_ dS)NrrSFrTr`p) r-r. apply_dropoutrr1 project_convr3r4r5 project_bnDropoutdropout)r:rrRr/r)rtrur;rrr.s z$EfficientNetFinalBlockLayer.__init__) embeddingsrZr>cCs0||}||}|jr,||}||}|Sr?)ryrzrxr|)r:r}rZrrrrA*s    z#EfficientNetFinalBlockLayer.forwardrBrCrDrErrfloatrfr.rFr\rGrArHrrr;rrrsrrc sJeZdZdZeeeeeeeeed fdd Ze j e j dddZ Z S)EfficientNetBlocka This corresponds to the expansion and depthwise convolution phase of each block in the original implementation. Args: config ([`EfficientNetConfig`]): Model configuration class. in_dim (`int`): Number of input channels. out_dim (`int`): Number of output channels. stride (`int`): Stride size to be used in convolution layers. expand_ratio (`int`): Expand ratio to set the output dimensions for the expansion and squeeze-excite layers. kernel_size (`int`): Kernel size for the depthwise convolution layer. drop_rate (`float`): Dropout rate to be used in the final phase of each block. id_skip (`bool`): Whether to apply dropout and sum the final hidden states with the input embeddings during the final phase of each block. Set to `True` for the first block of each stage. adjust_padding (`bool`): Whether to apply padding to only right and bottom side of the input kernel before the depthwise convolution operation, set to `True` for inputs with odd input sizes. ) rrRr/r) expand_ratiorrtrur_c st||_|jdkrdnd|_||} |jrDt||| |d|_t||jrR| n|||| d|_t||| |jd|_ t ||jr| n|||||d|_ dS)NrTFrQr^rhrs) r-r.rrjrP expansionr]rbrgsqueeze_exciterr projection) r:rrRr/r)rrrtrur_Z expand_in_dimr;rrr.Ps@   zEfficientNetBlock.__init__rYcCs<|}|jdkr||}||}||}|||}|S)Nr)rrrbrr)r:rZr}rrrrAys     zEfficientNetBlock.forwardr~rrr;rr5s)rcsFeZdZdZedfdd Zd ejee ee e ddd Z Z S) EfficientNetEncoderz Forward propogates the embeddings through each EfficientNet block. Args: config ([`EfficientNetConfig`]): Model configuration class. r#c s~t|_|j_fddt|j}tfdd|jD}d}g}t|D]}t ||j|}t ||j |}|j |} |j |} |j |} t|j|D]} | dkrdnd} | dkrdn| } | dkr|n|}||jkrdnd}|j||}t|||| | | || |d }|||d7}qqVt|_tj|t |d dd dd _tj|j|j|jd _t|j_dS)Ncsttj|Sr?)rmathceildepth_coefficient)Zrepeats)r:rr round_repeatssz3EfficientNetEncoder.__init__..round_repeatsc3s|]}|VqdSr?r).0n)rrr sz/EfficientNetEncoder.__init__..rTFr) rrRr/r)rrrtrur_irSrTr`)r-r.rrlenrKsumZnum_block_repeatsrangerrLstridesZ kernel_sizesZ expand_ratiosZdepthwise_paddingZdrop_connect_raterappendrZ ModuleListblocksr1top_convr3 hidden_dimr4r5top_bnr r7top_activation)r:rZnum_base_blocksZ num_blocksZcurr_block_numrirRr/r)rrjrur_rtblockr;)rr:rr.s^          zEfficientNetEncoder.__init__FT)rZoutput_hidden_states return_dictr>cCst|r |fnd}|jD]}||}|r||f7}q||}||}||}|shtdd||fDSt||dS)Ncss|]}|dk r|VqdSr?r)rvrrrrsz.EfficientNetEncoder.forward..)last_hidden_staterZ)rrrrtupler )r:rZrrZall_hidden_statesrrrrrAs     zEfficientNetEncoder.forward)FT) rBrCrDrErr.rFr\rrfr rArHrrr;rrs:rc@s2eZdZdZeZdZdZdZddZ d dd Z d S) EfficientNetPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. efficientnetr=TcCsjt|tjtjfr@|jjjd|jjd|j dk rf|j j n&t|tj rf|j j |jj ddS)zInitialize the weightsg)meanZstdNg?) r rLinearr1weightdataZnormal_rZinitializer_ranger*Zzero_Z LayerNormZfill_)r:modulerrr _init_weightss   z)EfficientNetPreTrainedModel._init_weightsFcCst|tr||_dSr?)r rZgradient_checkpointing)r:rvaluerrr_set_gradient_checkpointings z7EfficientNetPreTrainedModel._set_gradient_checkpointingN)F) rBrCrDrEr config_classZbase_model_prefixZmain_input_nameZsupports_gradient_checkpointingrrrrrrrs rzUThe bare EfficientNet model outputting raw features without any specific head on top.c sdeZdZedfdd Zeeeee e de dd e j eeeeeee fddd ZZS) EfficientNetModelr#cs~t|||_t||_t||_|jdkrDtj |j dd|_ n.|jdkrbtj |j dd|_ nt d|j|dS)NrT)Z ceil_moderz2config.pooling must be one of ['mean', 'max'] got )r-r.rr"r}rencoderZ pooling_typerZ AvgPool2drpoolerZ MaxPool2d ValueErrorZpooling post_initr9r;rrr.s     zEfficientNetModel.__init__Zvision) checkpoint output_typerZmodalityexpected_outputN)r=rrr>cCs|dk r |n|jj}|dk r |n|jj}|dkr8td||}|j|||d}|d}||}||jdd}|s||f|ddSt |||j dS)Nz You have to specify pixel_valuesrrrrr)r pooler_outputrZ) rruse_return_dictrr}rrZreshapeshaper rZ)r:r=rrZembedding_outputZencoder_outputsr pooled_outputrrrrAs*  zEfficientNetModel.forward)NNN)rBrCrDrr.rEFFICIENTNET_INPUTS_DOCSTRINGr_CHECKPOINT_FOR_DOCr _CONFIG_FOR_DOC_EXPECTED_OUTPUT_SHAPErFr\rrfrrrArHrrr;rrs$  rz EfficientNet Model with an image classification head on top (a linear layer on top of the pooled features), e.g. for ImageNet. c sdeZdZfddZeeeeee e dde j e e je ee eeeefdddZZS) "EfficientNetForImageClassificationcsdt||j|_||_t||_tj|jd|_ |jdkrNt |j |jnt |_ |dS)Nrvr)r-r. num_labelsrrrrr{Z dropout_rater|rrZIdentity classifierrr9r;rrr.Gs  $z+EfficientNetForImageClassification.__init__)rrrrN)r=labelsrrr>c Csv|dk r |n|jj}|j|||d}|r.|jn|d}||}||}d}|dk r6|jjdkr|jdkrxd|j_n4|jdkr|jt j ks|jt j krd|j_nd|j_|jjdkrt } |jdkr| | | }n | ||}nN|jjdkrt} | |d|j|d}n|jjdkr6t} | ||}|sf|f|dd} |dk rb|f| S| St|||jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NrrZ regressionZsingle_label_classificationZmulti_label_classificationr)losslogitsrZ)rrrrr|rZ problem_typerZdtyperFlongrrrlrviewrr rZ) r:r=rrroutputsrrrZloss_fctoutputrrrrASs@      "     z*EfficientNetForImageClassification.forward)NNNN)rBrCrDr.rrr_IMAGE_CLASS_CHECKPOINTr r_IMAGE_CLASS_EXPECTED_OUTPUTrFr\rZ LongTensorrfrrrArHrrr;rr?s&  r)T)8rErtypingrrrrFZtorch.utils.checkpointrZtorch.nnrrrZ activationsr Zmodeling_outputsr r r Zmodeling_utilsrutilsrrrrZconfiguration_efficientnetrZ get_loggerrBloggerrrrrrZ*EFFICIENTNET_PRETRAINED_MODEL_ARCHIVE_LISTZEFFICIENTNET_START_DOCSTRINGrrrrfr!Moduler"r1rIrPr]rgrrrrrrrrrrrsT       ''!QZ?