U ,-eV@sdZddlmZmZmZddlZddlZddlmZddlm Z m Z m Z ddl m Z ddlmZmZmZmZdd lmZdd lmZmZmZmZmZdd lmZd d lmZee Z!dZ"dZ#d dddgZ$dZ%dZ&dgZ'd1ej(e)e*ej(dddZ+Gdddej,Z-Gdddej,Z.Gdddej,Z/Gdddej,Z0Gd d!d!ej,Z1Gd"d#d#ej,Z2Gd$d%d%eZ3d&Z4d'Z5ed(e4Gd)d*d*e3Z6ed+e4Gd,d-d-e3Z7ed.e4Gd/d0d0e3eZ8dS)2z PyTorch ConvNext model.)OptionalTupleUnionN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BackboneOutputBaseModelOutputWithNoAttention(BaseModelOutputWithPoolingAndNoAttention$ImageClassifierOutputWithNoAttention)PreTrainedModel)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) BackboneMixin)ConvNextConfigrzfacebook/convnext-tiny-224iztabby, tabby catF)input drop_probtrainingreturncCsd|dks |s|Sd|}|jdfd|jd}|tj||j|jd}||||}|S)aF Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). Comment by Ross Wightman: This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use 'survival rate' as the argument. rrr)r)dtypedevice)shapendimtorchZrandrrZfloor_div)rrrZ keep_probr Z random_tensoroutputr%o/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/models/convnext/modeling_convnext.py drop_path@s r'csPeZdZdZd eeddfdd ZejejdddZ e d d d Z Z S) ConvNextDropPathzXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).N)rrcst||_dSN)super__init__r)selfr __class__r%r&r+Xs zConvNextDropPath.__init__ hidden_statesrcCst||j|jSr))r'rrr,r0r%r%r&forward\szConvNextDropPath.forward)rcCs d|jS)Nzp={})formatr)r,r%r%r& extra_repr_szConvNextDropPath.extra_repr)N) __name__ __module__ __qualname____doc__rfloatr+r"Tensorr2strr4 __classcell__r%r%r-r&r(Usr(cs6eZdZdZd fdd ZejejdddZZS) ConvNextLayerNormaALayerNorm that supports two data formats: channels_last (default) or channels_first. The ordering of the dimensions in the inputs. channels_last corresponds to inputs with shape (batch_size, height, width, channels) while channels_first corresponds to inputs with shape (batch_size, channels, height, width). ư> channels_lastcs`ttt||_tt||_||_ ||_ |j dkrTt d|j |f|_ dS)N)r?channels_firstzUnsupported data format: ) r*r+r Parameterr"onesweightZzerosbiaseps data_formatNotImplementedErrornormalized_shape)r,rHrErFr-r%r&r+is  zConvNextLayerNorm.__init__)xrcCs|jdkr*tjj||j|j|j|j}n|jdkr|j }| }|j ddd}|| dj ddd}||t ||j}|j|d}|jddddf||jddddf}|S)Nr?r@rT)Zkeepdim)r)rFr"rZ functionalZ layer_normrHrCrDrErr9meanpowsqrtto)r,rIZ input_dtypeusr%r%r&r2ss    ,zConvNextLayerNorm.forward)r>r?) r5r6r7r8r+r"r:r2r<r%r%r-r&r=cs r=cs4eZdZdZfddZejejdddZZ S)ConvNextEmbeddingszThis class is comparable to (and inspired by) the SwinEmbeddings class found in src/transformers/models/swin/modeling_swin.py. csLttj|j|jd|j|jd|_t|jdddd|_ |j|_dS)Nr kernel_sizestrider>r@rErF) r*r+rConv2d num_channels hidden_sizesZ patch_sizepatch_embeddingsr= layernormr,configr-r%r&r+s zConvNextEmbeddings.__init__) pixel_valuesrcCs4|jd}||jkrtd||}||}|S)NrzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.)r rW ValueErrorrYrZ)r,r]rW embeddingsr%r%r&r2s    zConvNextEmbeddings.forward r5r6r7r8r+r" FloatTensorr:r2r<r%r%r-r&rQs rQcs6eZdZdZdfdd ZejejdddZZ S) ConvNextLayera3This corresponds to the `Block` class in the original implementation. There are two equivalent implementations: [DwConv, LayerNorm (channels_first), Conv, GELU,1x1 Conv]; all in (N, C, H, W) (2) [DwConv, Permute to (N, H, W, C), LayerNorm (channels_last), Linear, GELU, Linear]; Permute back The authors used (2) as they find it slightly faster in PyTorch. Args: config ([`ConvNextConfig`]): Model configuration class. dim (`int`): Number of input channels. drop_path (`float`): Stochastic depth rate. Default: 0.0. rcsttj||dd|d|_t|dd|_t|d||_t |j |_ td|||_ |j dkrtj|j t|dd nd|_|d krt|nt|_dS) Nrr )rSpaddinggroupsr>rErT)Z requires_gradr)r*r+rrVdwconvr=rZLinearpwconv1r Z hidden_actactpwconv2Zlayer_scale_init_valuerAr"rBlayer_scale_parameterr(Identityr')r,r\dimr'r-r%r&r+s  zConvNextLayer.__init__r/cCs||}||}|dddd}||}||}||}||}|jdk rZ|j|}|dddd}|||}|S)NrrJr r)rgZpermuterZrirjrkrlr')r,r0rrIr%r%r&r2s       zConvNextLayer.forward)rr`r%r%r-r&rbs rbcs6eZdZdZd fdd ZejejdddZZ S) ConvNextStageaConvNeXT stage, consisting of an optional downsampling layer + multiple residual blocks. Args: config ([`ConvNextConfig`]): Model configuration class. in_channels (`int`): Number of input channels. out_channels (`int`): Number of output channels. depth (`int`): Number of residual blocks. drop_path_rates(`List[float]`): Stochastic depth rates for each layer. rJNc st|ks|dkrBtt|dddtj|||d|_n t|_pXdg|tjfddt|D|_ dS) Nrr>r@rUrRrcsg|]}t|dqS))rnr')rb).0jr\drop_path_rates out_channelsr%r& sz*ConvNextStage.__init__..) r*r+rZ Sequentialr=rVdownsampling_layerrmrangelayers)r,r\ in_channelsrtrSrTdepthrsr-rrr&r+s   zConvNextStage.__init__r/cCs||}||}|Sr))rvrxr1r%r%r&r2s  zConvNextStage.forward)rJrJrJNr`r%r%r-r&ros rocsDeZdZfddZdejeeeeee e fdddZ Z S) ConvNextEncoderc stt|_ddtd|jt|j  |j D}|j d}t |j D]H}|j |}t||||dkrrdnd|j |||d}|j||}qPdS)NcSsg|] }|qSr%)tolist)rprIr%r%r&rusz,ConvNextEncoder.__init__..rrJr)ryrtrTrzrs)r*r+rZ ModuleListstagesr"ZlinspaceZdrop_path_ratesumZdepthssplitrXrwZ num_stagesroappend)r,r\rsZprev_chsiZout_chsstager-r%r&r+s$     zConvNextEncoder.__init__FT)r0output_hidden_states return_dictrcCsj|rdnd}t|jD]\}}|r,||f}||}q|rD||f}|s^tdd||fDSt||dS)Nr%css|]}|dk r|VqdSr)r%)rpvr%r%r& sz*ConvNextEncoder.forward..)last_hidden_stater0) enumerater}tupler )r,r0rrZall_hidden_statesrZ layer_moduler%r%r&r2s    zConvNextEncoder.forward)FT) r5r6r7r+r"rarboolrrr r2r<r%r%r-r&r{s  r{c@s2eZdZdZeZdZdZdZddZ d dd Z d S) ConvNextPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. convnextr]TcCsjt|tjtjfr@|jjjd|jjd|j dk rf|j j n&t|tj rf|j j |jj ddS)zInitialize the weightsr)rKZstdNg?) isinstancerrhrVrCdataZnormal_r\Zinitializer_rangerDZzero_ LayerNormZfill_)r,moduler%r%r& _init_weightss   z%ConvNextPreTrainedModel._init_weightsFcCst|tr||_dSr))rr{Zgradient_checkpointing)r,rvaluer%r%r&_set_gradient_checkpointing+s z3ConvNextPreTrainedModel._set_gradient_checkpointingN)F) r5r6r7r8r config_classZbase_model_prefixZmain_input_nameZsupports_gradient_checkpointingrrr%r%r%r&rs raJ This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`ConvNextConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. aF Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`ConvNextImageProcessor.__call__`] for details. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zQThe bare ConvNext model outputting raw features without any specific head on top.c s^eZdZfddZeeeeee de dd e j e ee eeeefdddZZS) ConvNextModelcsJt|||_t||_t||_tj|j d|j d|_ | dS)Nre) r*r+r\rQr_r{encoderrrrXZlayer_norm_epsrZ post_initr[r-r%r&r+Ns    zConvNextModel.__init__Zvision) checkpoint output_typerZmodalityexpected_outputNr]rrrcCs|dk r |n|jj}|dk r |n|jj}|dkr8td||}|j|||d}|d}||ddg}|s||f|ddSt|||j dS)Nz You have to specify pixel_valuesrrrrr)r pooler_outputr0) r\ruse_return_dictr^r_rrZrKr r0)r,r]rrembedding_outputZencoder_outputsr pooled_outputr%r%r&r2[s( zConvNextModel.forward)NNN)r5r6r7r+rCONVNEXT_INPUTS_DOCSTRINGr_CHECKPOINT_FOR_DOCr _CONFIG_FOR_DOC_EXPECTED_OUTPUT_SHAPEr"rarrrrr2r<r%r%r-r&rIs$   rz ConvNext Model with an image classification head on top (a linear layer on top of the pooled features), e.g. for ImageNet. c sdeZdZfddZeeeeee e dde j e e je ee eeeefdddZZS) ConvNextForImageClassificationcsRt||j|_t||_|jdkr 1` a classification loss is computed (Cross-Entropy). NrrZ regressionZsingle_label_classificationZmulti_label_classificationrrJ)losslogitsr0)r\rrrrZ problem_typerrr"longintrZsqueezerviewrrr0) r,r]rrroutputsrrrZloss_fctr$r%r%r&r2s>     "     z&ConvNextForImageClassification.forward)NNNN)r5r6r7r+rrr_IMAGE_CLASS_CHECKPOINTrr_IMAGE_CLASS_EXPECTED_OUTPUTr"rarZ LongTensorrrrr2r<r%r%r-r&rs&  rzQ ConvNeXt backbone, to be used with frameworks like DETR and MaskFormer. csPeZdZfddZeeeeedde j e e e e edddZ ZS) ConvNextBackbonecst|t|t||_t||_|jdg|j|_i}t |j |j D]\}}t |dd||<qRt ||_|dS)Nrr@)rF)r*r+Z_init_backbonerQr_r{rrXZ num_featureszipZ _out_featuresZchannelsr=rZ ModuleDicthidden_states_normsr)r,r\rrrWr-r%r&r+s     zConvNextBackbone.__init__)rrNrc Cs|dk r |n|jj}|dk r |n|jj}||}|j|ddd}|j}d}tt|jdd|ddD].\}\} } | |j krl|j | | } || f7}ql|s|f} |r| |jf7} | St ||r|jndddS)az Returns: Examples: ```python >>> from transformers import AutoImageProcessor, AutoBackbone >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> processor = AutoImageProcessor.from_pretrained("facebook/convnext-tiny-224") >>> model = AutoBackbone.from_pretrained("facebook/convnext-tiny-224") >>> inputs = processor(image, return_tensors="pt") >>> outputs = model(**inputs) ```NTrr%r) feature_mapsr0Z attentions) r\rrr_rr0rrZ stage_namesZ out_featuresrr ) r,r]rrrrr0ridxrZ hidden_stater$r%r%r&r2s2 ,    zConvNextBackbone.forward)NN)r5r6r7r+rrrr rr"r:rrr2r<r%r%r-r&rs  r)rF)9r8typingrrrr"Ztorch.utils.checkpointrZtorch.nnrrrZ activationsr Zmodeling_outputsr r r rZmodeling_utilsrutilsrrrrrZutils.backbone_utilsrZconfiguration_convnextrZ get_loggerr5loggerrrrrrZ&CONVNEXT_PRETRAINED_MODEL_ARCHIVE_LISTr:r9rr'Moduler(r=rQrbror{rZCONVNEXT_START_DOCSTRINGrrrrr%r%r%r&sX       , / ;L