U 9%e@sdZddlZddlZddlZddlmZddlmZm Z m Z ddl Z ddl Z ddl m Z ddlmZmZmZddlmZdd lmZdd lmZdd lmZmZmZdd lmZmZmZm Z m!Z!m"Z"dd l#m$Z$ddl%m&Z&e!'e(Z)dZ*dZ+dddgZ,dZ-dZ.dgZ/eGdddeZ0eGdddeZ1eGdddeZ2eGdddeZ3ddZ4dd Z5Gd!d"d"e j6Z7Gd#d$d$e j6Z8Gd%d&d&e j6Z9dNe j:e;eGd.d/d/e j6Z?Gd0d1d1e j6Z@Gd2d3d3e j6ZAGd4d5d5e j6ZBGd6d7d7e j6ZCGd8d9d9e j6ZDGd:d;d;e j6ZEGdd?d?eZGd@ZHdAZIedBeHGdCdDdDeGZJedEeHGdFdGdGeGZKedHeHGdIdJdJeGZLedKeHGdLdMdMeGe$ZMdS)Oz PyTorch Swin Transformer model.N) dataclass)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BackboneOutput)PreTrainedModel) find_pruneable_heads_and_indicesmeshgridprune_linear_layer) ModelOutputadd_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) BackboneMixin) SwinConfigrzµsoft/swin-tiny-patch4-window7-2241iztabby, tabby catc@sbeZdZUdZdZejed<dZe e ejed<dZ e e ejed<dZ e e ejed<dS)SwinEncoderOutputa Swin encoder's outputs, with potential hidden states and attentions. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each stage) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. reshaped_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, hidden_size, height, width)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs reshaped to include the spatial dimensions. Nlast_hidden_state hidden_states attentionsreshaped_hidden_states) __name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rrrrrr'r'e/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/transformers/models/swin/modeling_swin.pyrCs rc@steZdZUdZdZejed<dZe ejed<dZ e e ejed<dZ e e ejed<dZ e e ejed<dS)SwinModelOutputaT Swin model's outputs that also contains a pooling of the last hidden states. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`, *optional*, returned when `add_pooling_layer=True` is passed): Average pooling of the last layer hidden-state. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each stage) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. reshaped_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, hidden_size, height, width)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs reshaped to include the spatial dimensions. Nr pooler_outputrrr)r r!r"r#rr$r%r&r*rrrrrr'r'r'r(r)ds r)c@seZdZUdZdZeejed<dZ ejed<dZ ee ejed<dZ ee ejed<dZ ee ejed<edd ZdS) SwinMaskedImageModelingOutputa Swin masked image model outputs. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `bool_masked_pos` is provided): Masked image modeling (MLM) loss. reconstruction (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Reconstructed pixel values. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each stage) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. reshaped_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, hidden_size, height, width)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs reshaped to include the spatial dimensions. NlossreconstructionrrrcCstdt|jS)Nzlogits attribute is deprecated and will be removed in version 5 of Transformers. Please use the reconstruction attribute to retrieve the final output instead.)warningswarn FutureWarningr-selfr'r'r(logitss z$SwinMaskedImageModelingOutput.logits)r r!r"r#r,rr$r%r&r-rrrrpropertyr3r'r'r'r(r+s r+c@steZdZUdZdZeejed<dZ ejed<dZ ee ejed<dZ ee ejed<dZ ee ejed<dS)SwinImageClassifierOutputa Swin outputs for image classification. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification (or regression if config.num_labels==1) loss. logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Classification (or regression if config.num_labels==1) scores (before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each stage) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. reshaped_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, hidden_size, height, width)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs reshaped to include the spatial dimensions. Nr,r3rrr)r r!r"r#r,rr$r%r&r3rrrrr'r'r'r(r5s r5cCsR|j\}}}}|||||||||}|ddddddd|||}|S)z2 Partitions the given input into windows. rrr shapeviewpermute contiguous) input_feature window_size batch_sizeheightwidth num_channelswindowsr'r'r(window_partitions$rFcCsN|jd}|d|||||||}|ddddddd|||}|S)z? Merges windows to produce higher resolution features. r9rrr r6r7r8r:)rEr@rBrCrDr'r'r(window_reverses $rGcsHeZdZdZd fdd Zd eejeeje ej dddZ Z S) SwinEmbeddingszW Construct the patch and position embeddings. Optionally, also the mask token. Fcstt||_|jj}|jj|_|r@tt dd|j nd|_ |j rjtt d|d|j |_nd|_t|j |_t|j|_dS)Nr)super__init__SwinPatchEmbeddingspatch_embeddings num_patches grid_size patch_gridr Parameterr$zeros embed_dim mask_tokenZuse_absolute_embeddingsposition_embeddings LayerNormnormDropouthidden_dropout_probdropout)r2configuse_mask_tokenrM __class__r'r(rJs    zSwinEmbeddings.__init__N) pixel_valuesbool_masked_posreturnc Cs||\}}||}|\}}}|dk rb|j||d}|d|} |d| || }|jdk rv||j}||}||fS)Nr9?) rLrVsizerSexpand unsqueezeZtype_asrTrY) r2r^r_ embeddingsoutput_dimensionsrAZseq_len_Z mask_tokensmaskr'r'r(forwards    zSwinEmbeddings.forward)F)N) r r!r"r#rJrr$r% BoolTensorrTensorri __classcell__r'r'r\r(rHsrHcsLeZdZdZfddZddZeeje ej e e fdddZ Z S) rKz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. cst|j|j}}|j|j}}t|tjj r8|n||f}t|tjj rR|n||f}|d|d|d|d}||_||_||_||_ |d|d|d|df|_ t j ||||d|_dS)Nrr) kernel_sizeZstride)rIrJ image_size patch_sizerDrR isinstance collectionsabcIterablerMrNrConv2d projection)r2rZrnrorD hidden_sizerMr\r'r(rJ s  "zSwinPatchEmbeddings.__init__cCs||jddkr