U ,-eM/@sdZddlZddlmZddlmZmZmZmZm Z ddl Z ddl Z ddl Z ddl mZmZddlmZddlmZmZdd lmZdd lmZmZmZmZmZmZd d lmZm Z m!Z!erdd l"m#Z#e$e%Z&dZ'dddgZ(dQe je j)ee*dddZ+e je jdddZ,e je jdddZ-eGdddeZ.eedddZ/eedd d!Z0d"d#Z1d$d%Z2eGd&d'd'eZ3eGd(d)d)eZ4Gd*d+d+ej5Z6Gd,d-d-ej5Z7Gd.d/d/ej5Z8Gd0d1d1ej5Z9Gd2d3d3ej5Z:Gd4d5d5eZ;d6Zd9Z?d:Z@d;ZAGdd?d@ZEGdAdBdBej5ZFGdCdDdDe;ZGGdEdFdFej5ZHGdGdHdHe;ZIee<GdIdJdJe;ZJGdKdLdLej5ZKGdMdNdNej5ZLGdOdPdPe;ZMdS)Sz PyTorch OWL-ViT model.N) dataclass)AnyDictOptionalTupleUnion)Tensornn)ACT2FN)BaseModelOutputBaseModelOutputWithPooling)PreTrainedModel) ModelOutputadd_start_docstrings%add_start_docstrings_to_model_forwardis_vision_availableloggingreplace_return_docstrings) OwlViTConfigOwlViTTextConfigOwlViTVisionConfig)center_to_corners_formatzgoogle/owlvit-base-patch32zgoogle/owlvit-base-patch16zgoogle/owlvit-large-patch14)maskdtypetgt_lencCsj|\}}|dk r|n|}|ddddddf|d|||}d|}||tjt|jS)z_ Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`. Nr?)sizeexpandtoZ masked_filltorchboolfinfomin)rrrbszsrc_lenZ expanded_maskZ inverted_maskr'k/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/models/owlvit/modeling_owlvit.py _expand_mask:s  *r))logitsreturncCstj|tjt||jdS)Ndevice)r functionalZ cross_entropyr!arangelenr-)r*r'r'r(contrastive_lossIsr1) similarityr+cCs t|}t|}||dS)Ng@)r1t)r2Z caption_lossZ image_lossr'r'r( owlvit_lossNs r4c@seZdZUdZdZeejed<dZ ejed<dZ ejed<dZ ejed<dZ ejed<dZ eed<dZeed <eed d d ZdS) OwlViTOutputa% Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds (`torch.FloatTensor` of shape `(batch_size * num_max_text_queries, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`OwlViTTextModel`]. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`OwlViTVisionModel`]. text_model_output (Tuple[`BaseModelOutputWithPooling`]): The output of the [`OwlViTTextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`OwlViTVisionModel`]. Nlosslogits_per_imagelogits_per_text text_embeds image_embedstext_model_outputvision_model_outputr+cstfddDS)Nc3s,|]$}|dkr|n t|VqdS)r;r<Ngetattrto_tuple.0kselfr'r( tsz(OwlViTOutput.to_tuple..tuplekeysrEr'rEr(rAss zOwlViTOutput.to_tuple)__name__ __module__ __qualname____doc__r6rr! FloatTensor__annotations__r7r8r9r:r;r r<rrrAr'r'r'r(r5Ts   r5)r3r+cCsH|r&|jtjtjfkr|S|S|jtjtjfkr<|S|SdSN) Zis_floating_pointrr!float32float64floatZint32Zint64int)r3r'r'r(_upcast{srV)boxesr+cCsHt|}|dddf|dddf|dddf|dddfS)a Computes the area of a set of bounding boxes, which are specified by its (x1, y1, x2, y2) coordinates. Args: boxes (`torch.FloatTensor` of shape `(number_of_boxes, 4)`): Boxes for which the area will be computed. They are expected to be in (x1, y1, x2, y2) format with `0 <= x1 < x2` and `0 <= y1 < y2`. Returns: `torch.FloatTensor`: a tensor containing the area for each box. Nrr r)rV)rWr'r'r(box_areas rYc Cst|}t|}t|dddddf|ddddf}t|dddddf|ddddf}||jdd}|dddddf|dddddf}|dddf||}||} | |fS)NrXrr$r)rYr!maxr$clamp) boxes1boxes2Zarea1Zarea2Zleft_topZ right_bottom width_heightZinterunioniour'r'r(box_ious..,rbcCs*|ddddf|ddddfks:td||ddddf|ddddfksttd|t||\}}t|dddddf|ddddf}t|dddddf|ddddf}||jdd}|dddddf|dddddf}||||S)z Generalized IoU from https://giou.stanford.edu/. The boxes should be in [x0, y0, x1, y1] (corner) format. Returns: `torch.FloatTensor`: a [N, M] pairwise matrix, where N = len(boxes1) and M = len(boxes2) NrXzr?rBrEr'r(rGsz7OwlViTObjectDetectionOutput.to_tuple..rHrEr'rEr(rAs z$OwlViTObjectDetectionOutput.to_tuple)rKrLrMrNr6rr!rOrPrirr*rjr9r:rkr;r r<rrrAr'r'r'r(rhs   rhc@seZdZUdZdZejed<dZejed<dZ ejed<dZ ejed<dZ ejed<dZ ejed<dZ eed <dZeed <eed d d ZdS)&OwlViTImageGuidedObjectDetectionOutputa Output type of [`OwlViTForObjectDetection.image_guided_detection`]. Args: logits (`torch.FloatTensor` of shape `(batch_size, num_patches, num_queries)`): Classification logits (including no-object) for all queries. target_pred_boxes (`torch.FloatTensor` of shape `(batch_size, num_patches, 4)`): Normalized boxes coordinates for all queries, represented as (center_x, center_y, width, height). These values are normalized in [0, 1], relative to the size of each individual target image in the batch (disregarding possible padding). You can use [`~OwlViTImageProcessor.post_process_object_detection`] to retrieve the unnormalized bounding boxes. query_pred_boxes (`torch.FloatTensor` of shape `(batch_size, num_patches, 4)`): Normalized boxes coordinates for all queries, represented as (center_x, center_y, width, height). These values are normalized in [0, 1], relative to the size of each individual query image in the batch (disregarding possible padding). You can use [`~OwlViTImageProcessor.post_process_object_detection`] to retrieve the unnormalized bounding boxes. image_embeds (`torch.FloatTensor` of shape `(batch_size, patch_size, patch_size, output_dim`): Pooled output of [`OwlViTVisionModel`]. OWL-ViT represents images as a set of image patches and computes image embeddings for each patch. query_image_embeds (`torch.FloatTensor` of shape `(batch_size, patch_size, patch_size, output_dim`): Pooled output of [`OwlViTVisionModel`]. OWL-ViT represents images as a set of image patches and computes image embeddings for each patch. class_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`): Class embeddings of all image patches. OWL-ViT represents images as a set of image patches where the total number of patches is (image_size / patch_size)**2. text_model_output (Tuple[`BaseModelOutputWithPooling`]): The output of the [`OwlViTTextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`OwlViTVisionModel`]. Nr*r:query_image_embedstarget_pred_boxesquery_pred_boxesrkr;r<r=cstfddDS)Nc3s,|]$}|dkr|n t|VqdSr>r?rBrEr'r(rGszBOwlViTImageGuidedObjectDetectionOutput.to_tuple..rHrEr'rEr(rAs z/OwlViTImageGuidedObjectDetectionOutput.to_tuple)rKrLrMrNr*r!rOrPr:rmrnrorkr;r r<rrrAr'r'r'r(rls   rlcs6eZdZedfdd ZejejdddZZ S)OwlViTVisionEmbeddingsconfigcst||_|j|_tt|j|_ tj |j |j|j |j dd|_ |j|j d|_|jd|_t|j|j|_|jdt|jddddS)NF)Z in_channelsZ out_channelsZ kernel_sizeZstridebiasrXr position_idsr persistent)super__init__rr hidden_size embed_dimr Parameterr!Zrandnclass_embeddingZConv2dZ num_channelsZ patch_sizepatch_embeddingZ image_size num_patchesZ num_positions Embeddingposition_embeddingregister_bufferr/rrFrr __class__r'r(rz"s  zOwlViTVisionEmbeddings.__init__) pixel_valuesr+cCs\|jd}||}|ddd}|j|dd}tj||gdd}|||j }|S)NrrXrrvdim) shaperflatten transposer~rr!catrrt)rFr batch_sizeZ patch_embedsrk embeddingsr'r'r(forward5s  zOwlViTVisionEmbeddings.forward) rKrLrMrrzr!rOrr __classcell__r'r'rr(rp!srpcsLeZdZedfdd Zdeejeejeejej dddZ Z S) OwlViTTextEmbeddingsrqcsPtt|j|j|_t|j|j|_|j dt |j ddddS)NrtruFrw) ryrzr rZ vocab_sizer{token_embeddingZmax_position_embeddingsrrr!r/rrrr'r(rzBs zOwlViTTextEmbeddings.__init__N) input_idsrt inputs_embedsr+cCsb|dk r|jdn|jd}|dkr:|jddd|f}|dkrL||}||}||}|S)Nrv)rrtrr)rFrrtrZ seq_lengthZposition_embeddingsrr'r'r(rLs  zOwlViTTextEmbeddings.forward)NNN) rKrLrMrrzrr! LongTensorrOrrrr'r'rr(rAs rc szeZdZdZfddZejeedddZd eje eje eje e e eje eje e ejfd d d Z Z S) OwlViTAttentionz=Multi-headed attention from 'Attention Is All You Need' papercst||_|j|_|j|_|j|j|_|j|j|jkrZtd|jd|jd|jd|_ |j |_ t |j|j|_t |j|j|_t |j|j|_t |j|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).)ryrzrrr{r|Znum_attention_heads num_headshead_dimrdscaleZattention_dropoutdropoutr Lineark_projv_projq_projout_projrrr'r(rzcs  zOwlViTAttention.__init__)tensorseq_lenr%cCs ||||j|jddS)NrrX)viewrrr contiguous)rFrrr%r'r'r(_shapevszOwlViTAttention._shapeNF hidden_statesattention_maskcausal_attention_maskoutput_attentionsr+cCs|\}}}|||j}|||d|} |||d|} ||jd|jf} ||||j| }| j| } | j| } | d} t || dd} | ||j|| fkrt d||j|| fd| |dk rD||d|| fkrt d|d|| fd|| ||j|| |} | ||j|| } |dk r||d|| fkrt d|d|| fd|| ||j|| |} | ||j|| } t jj| dd} |r| ||j|| }|||j|| } nd}t jj| |j|jd }|| j}t || }|||j||jfkr^t d ||j||jfd||||j||j}| dd}||||}||}||fS) z#Input shape: Batch x Time x ChannelrvrrXz$Attention weights should be of size z , but is Nz!Attention mask should be of size r)ptrainingz `attn_output` should be of size )rrrrrrrrrr!Zbmmrrdr r.Zsoftmaxrrr rreshaper)rFrrrrr%rr|Z query_statesZ key_statesZ value_statesZ proj_shaper& attn_weightsZattn_weights_reshapedZ attn_probsZ attn_outputr'r'r(rysZ          zOwlViTAttention.forward)NNF)rKrLrMrNrzr!rrUrrr"rrrr'r'rr(r`s rcs0eZdZfddZejejdddZZS) OwlViTMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSrQ) ryrzrrr Z hidden_act activation_fnr rr{Zintermediate_sizefc1fc2rrr'r(rzs   zOwlViTMLP.__init__)rr+cCs"||}||}||}|SrQ)rrr)rFrr'r'r(rs   zOwlViTMLP.forward)rKrLrMrzr!rrrr'r'rr(rs rcsJeZdZedfdd Zdejejejeee ej dddZ Z S) OwlViTEncoderLayerrqcsRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSN)Zeps) ryrzr{r|r self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2rrr'r(rzs    zOwlViTEncoderLayer.__init__FrcCsd|}||}|j||||d\}}||}|}||}||}||}|f}|r`||f7}|S)aI Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. `(config.encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rrrr)rrrr)rFrrrrZresidualroutputsr'r'r(rs"     zOwlViTEncoderLayer.forward)F) rKrLrMrrzr!rrr"rrOrrr'r'rr(rs rc@s4eZdZdZeZdZdZdgZddZ d dd Z d S) OwlViTPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. owlvitTrcCsX|jj}t|trF|jjjjd|dd|jjjjd|ddnt|t r|jj}t j j|j d|j d|dt j j|jj|jj|dt j j|jj|jj|dnTt|trD|jj}|j dd|jjd|}|j d|}t j j|jj|dt j j|jj|dt j j|jj|dt j j|jj|dnt|tr|jj}|jjdd|jjd|}d|jjd|}t j j|jj|dt j j|jj|dnPt|trt j j|jj|jd|jjdt j j|jj|jd|jjdt|t jr.|j j!|jj"dt|t j#rT|j dk rT|j j!dS) zInitialize the weightsg{Gz?)meanstdr)rrXrN)$rrZinitializer_factor isinstancerrweightdataZnormal_rrpr initr~r|rZinitializer_rangernum_hidden_layersrrrrrr{rr OwlViTModeltext_projectiontext_embed_dimvisual_projectionvision_embed_dimrrsZzero_Zfill_r)rFmodulefactorZ in_proj_stdZ out_proj_stdZfc_stdr'r'r( _init_weightssL       z#OwlViTPreTrainedModel._init_weightsFcCst|tr||_dSrQ)r OwlViTEncodergradient_checkpointing)rFrvaluer'r'r(_set_gradient_checkpointingCs z1OwlViTPreTrainedModel._set_gradient_checkpointingN)F) rKrLrMrNr config_classZbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesrrr'r'r'r(rs*raP Parameters: This model is a PyTorch [torch.nn.Module](https: //pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. config ([`OwlViTConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `(batch_size * num_max_text_queries, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, num_max_text_queries, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. input_ids (`torch.LongTensor` of shape `(batch_size * num_max_text_queries, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids). attention_mask (`torch.Tensor` of shape `(batch_size, num_max_text_queries, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_hidden_states (`bool`, *optional*): Whether or not to return the last hidden state. See `text_model_last_hidden_state` and `vision_model_last_hidden_state` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a_ Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. query_pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values of query image(s) to be detected. Pass in one query image per target image. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. c s`eZdZdZedfdd Zd eejeejee ee ee e e e fdddZ ZS) rz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`OwlViTEncoderLayer`]. Args: config: OwlViTConfig rqcs4ttfddtjD|_d|_dS)Ncsg|] }tqSr')r)rC_rqr'r( sz*OwlViTEncoder.__init__..F)ryrzr Z ModuleListrangerlayersrrrrqr(rzs  zOwlViTEncoder.__init__N)rrroutput_hidden_states return_dictr+c sdk r n|jj|dk r |n|jj}|dk r4|n|jj}|rDdnd}rPdnd}|} |jD]n} |rp|| f}|jr|jrfdd} tjj | | | ||} n| | ||d} | d} r^|| df}q^|r|| f}|st dd | ||fDSt | ||d S) a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`). attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr'csfdd}|S)Ncs|fSrQr')inputs)rrr'r(custom_forwardszLOwlViTEncoder.forward..create_custom_forward..custom_forwardr')rrr)rr(create_custom_forwardsz4OwlViTEncoder.forward..create_custom_forwardrrrcss|]}|dk r|VqdSrQr')rCvr'r'r(rG sz(OwlViTEncoder.forward..)last_hidden_stater attentions) rrrruse_return_dictrrrr!utils checkpointrIr ) rFrrrrrrZencoder_statesZall_attentionsrZ encoder_layerrZ layer_outputsr'rr(rsH       zOwlViTEncoder.forward)NNNNN)rKrLrMrNrrzrr!rr"rrr rrr'r'rr(rs r)input_ids_shaperr-past_key_values_lengthcCs|\}}tj||ft|j|d}tj|d|d}|||d|ddkd||}|dkrtj tj ||||d|gdd}|ddddddf |d|||S)zB Make causal mask used for bi-directional self-attention. r,rvrr)rr-rN) r!fullr#r$r/rZ masked_fill_rr rzerosr)rrr-rr%rrZ mask_condr'r'r(_make_causal_masks"  rc steZdZedfdd Zeeeeedd e j e e j e e j e e e e e e e eefdddZZS) OwlViTTextTransformerrqcs@t||_|j}t||_t||_tj ||j d|_ dSr) ryrzrrr{rrrencoderr rrfinal_layer_norm)rFrrr|rr'r(rz&s    zOwlViTTextTransformer.__init__ output_typerN)rrrtrrrr+c Cs|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|}|d|d}|j||d}t||j|j d} |dk rt ||j}|j ||| |||d} | d} | | } | t j| jd| j d|t jjdd| j f} |s| | f| ddSt| | | j| jd S)  Returns: Nrv)rrtr,)rrrrrrrrrrZ pooler_outputrr)rrrrrrrrrrr-r)rrr!r/rr rUZargmaxr rr) rFrrrtrrrZ input_shaperrencoder_outputsr pooled_outputr'r'r(r.sB   zOwlViTTextTransformer.forward)NNNNN)rKrLrMrrzrOWLVIT_TEXT_INPUTS_DOCSTRINGrr r!rrr"rrrrr'r'rr(r%s"  rc seZdZeZedfdd ZejdddZddZ e e e e ed dejeejeeeeeeeee fd d d ZZS)OwlViTTextModelrqcs"t|t||_|dSrQ)ryrzr text_model post_initrrr'r(rzps  zOwlViTTextModel.__init__r=cCs |jjjSrQrrrrEr'r'r(get_input_embeddingsvsz$OwlViTTextModel.get_input_embeddingscCs||jj_dSrQr)rFrr'r'r(set_input_embeddingsysz$OwlViTTextModel.set_input_embeddingsrNrrrrrr+cCs|j|||||dS)ay Returns: Examples: ```python >>> from transformers import AutoProcessor, OwlViTTextModel >>> model = OwlViTTextModel.from_pretrained("google/owlvit-base-patch32") >>> processor = AutoProcessor.from_pretrained("google/owlvit-base-patch32") >>> inputs = processor( ... text=[["a photo of a cat", "a photo of a dog"], ["photo of a astranaut"]], return_tensors="pt" ... ) >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled (EOS token) states ```rrrrr)r)rFrrrrrr'r'r(r|szOwlViTTextModel.forward)NNNN)rKrLrMrrrzr Modulerrrrrr r!rrr"rrrrr'r'rr(rms$  rc sdeZdZedfdd Zeeeeedd e j e e e e e e e eefdddZZS) OwlViTVisionTransformerrqcsPt||_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSr) ryrzrrrprr rr{r pre_layernormrrpost_layernormrrr'r(rzs    z OwlViTVisionTransformer.__init__rNrrrrr+c Cs|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|jjjj}||}||}| |}|j ||||d}|d}|dddddf} | | } |s|| f|ddSt || |j |jdS)rN)rrrrrrr)rrrrrrrrrr rrrr rr) rFrrrrZexpected_input_dtyperrrrr'r'r(rs2      zOwlViTVisionTransformer.forward)NNN)rKrLrMrrzrOWLVIT_VISION_INPUTS_DOCSTRINGrr r!rOrr"rrrrr'r'rr(rs   rc seZdZeZdZedfdd ZejdddZ e e e e edd eejeeeeeeeee fd d d ZZS)OwlViTVisionModelrrqcs"t|t||_|dSrQ)ryrzr vision_modelrrrr'r(rzs  zOwlViTVisionModel.__init__r=cCs |jjjSrQ)rrrrEr'r'r(rsz&OwlViTVisionModel.get_input_embeddingsrNrcCs|j||||dS)a Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, OwlViTVisionModel >>> model = OwlViTVisionModel.from_pretrained("google/owlvit-base-patch32") >>> processor = AutoProcessor.from_pretrained("google/owlvit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled CLS states ```rrrr)r)rFrrrrr'r'r(rs zOwlViTVisionModel.forward)NNNN)rKrLrMrrZmain_input_namerzr rrrrrr rr!rOr"rrrrr'r'rr(rs"  rcseZdZeZedfdd Zeedee j ee j ee ee ee e j dddZ eedee j ee ee ee e j dd d Zeeeeed dee jee j ee j ee ee ee ee ee eeefd d dZZS)rrqcst|t|jts.tdt|jdt|jtsPtdt|jd|j}|j}|j |_ |j |_ |j |_ t ||_t||_tj|j |j dd|_tj|j |j dd|_tt|j|_|dS)NzMconfig.text_config is expected to be of type OwlViTTextConfig but is of type .zQconfig.vision_config is expected to be of type OwlViTVisionConfig but is of type F)rs)ryrzr text_configrrdtype vision_configrZprojection_dimr{rrrrrrr rrrr}r!rZlogit_scale_init_value logit_scaler)rFrrrrrr'r(rzs(     zOwlViTModel.__init__Nrc Cs:|dk r |n|jj}|j|||d}|d}||}|S)a Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`OwlViTTextModel`]. Examples: ```python >>> from transformers import AutoProcessor, OwlViTModel >>> model = OwlViTModel.from_pretrained("google/owlvit-base-patch32") >>> processor = AutoProcessor.from_pretrained("google/owlvit-base-patch32") >>> inputs = processor( ... text=[["a photo of a cat", "a photo of a dog"], ["photo of a astranaut"]], return_tensors="pt" ... ) >>> text_features = model.get_text_features(**inputs) ```N)rrrr)rrrrr) rFrrrrrZ text_outputrZ text_featuresr'r'r(get_text_features1s  zOwlViTModel.get_text_featuresrcCsd|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|j||||d}|d}||}|S)aB Returns: image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`OwlViTVisionModel`]. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, OwlViTModel >>> model = OwlViTModel.from_pretrained("google/owlvit-base-patch32") >>> processor = AutoProcessor.from_pretrained("google/owlvit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> image_features = model.get_image_features(**inputs) ```Nrr)rrrrrrr)rFrrrrvision_outputsrimage_featuresr'r'r(get_image_featuresTs zOwlViTModel.get_image_featuresr) rrr return_lossrrreturn_base_image_embedsrr+c  Csd|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|j||||d} |j|||||d} | d} || } | d} || } | tj j | dddd} | tj j | dddd} |j | j}t| | |}|}d}|rt|}|rtd t| d }|j|} n| } |sN||| | | | f}|dk rJ|f|S|St|||| | | | d S) a[ Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, OwlViTModel >>> model = OwlViTModel.from_pretrained("google/owlvit-base-patch32") >>> processor = AutoProcessor.from_pretrained("google/owlvit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(text=[["a photo of a cat", "a photo of a dog"]], images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities ```NrrrrXrvT)ordrkeepdimz`return_base_image_embeds` is deprecated and will be removed in v4.27 of Transformers, one can obtain the base (unprojected) image embeddings from outputs.vision_model_output.r)r6r7r8r9r:r;r<)rrrrrrrrrr!linalgnormr expr r-matmulr3r4warningswarn FutureWarningrr5)rFrrrrrrrrr  text_outputsr9r:Ztext_embeds_normr r8r7r6routputr'r'r(rsb   zOwlViTModel.forward)NNNNN)NNNN)NNNNNNNN)rKrLrMrrrzrrrr!rr"rOr rr OWLVIT_INPUTS_DOCSTRINGrr5rrrrrr'r'rr(r sb ",  rcs6eZdZedfdd ZejejdddZZ S)OwlViTBoxPredictionHeadrqcsJt|jj}t|||_t|||_t|_ t|d|_ dS)N) ryrzrr{r rdense0dense1ZGELUgeludense2)rFrrwidthrr'r(rzs   z OwlViTBoxPredictionHead.__init__)r r+cCs6||}||}||}||}||}|SrQ)rr rr!)rFr rr'r'r(rs      zOwlViTBoxPredictionHead.forward) rKrLrMrrzr!rrOrrr'r'rr(rs rcsJeZdZedfdd Zejeejeeje ejdddZ Z S)OwlViTClassPredictionHeadrqcsZt|jj}|jj|_t|j||_t|jd|_ t|jd|_ t |_ dS)Nr) ryrzrr{r query_dimr rr logit_shiftr ZELUelu)rFrrZout_dimrr'r(rzs  z"OwlViTClassPredictionHead.__init__)r: query_embeds query_maskr+c Cs||}|dkrJ|j}|jdd\}}t|||jf|}||fS|tjj|dddd}|tjj|dddd}t d||}| |} | |} | | d} || | }|dk r|j dkrtj|dd }|tj}t|d kd |}|tj}||fS) NrXrvT)rrgư>z...pd,...qd->...pqrrrrg.)rr-rr!rr$r rreinsumr%r r&ndimZ unsqueezerSwhererR) rFr:r'r(image_class_embedsr-rr pred_logitsr%r r'r'r(rs(        z!OwlViTClassPredictionHead.forward) rKrLrMrrzr!rOrrrrrr'r'rr(r#s  r#c seZdZeZedfdd ZejdddZejejddd Z ejejejd d d Z d!eje eje ej e ejdddZd"ej ejej e ee ee ejdddZd#eje ee ee ejdddZejejejdddZeeeeedd$eje eje ee ee eedddZeeeeedd%ej eje ej e ee ee eeddd ZZS)&OwlViTForObjectDetectionrqcsPt|t||_t||_t||_tj |j j |j j d|_ t|_dSr)ryrzrrr# class_headrbox_headr rrr{r layer_normZSigmoidsigmoidrrr'r(rz+s     z!OwlViTForObjectDetection.__init__) feature_mapc Cs|jdkstd|j}|jd}tjttd|dtd|dddtj }|t ||gtj }| |jd|jd|jd}t ||}|S)NrzJExpected input shape is [batch_size, num_patches, num_patches, hidden_dim]rrvZaxisrrX)r*rdr-rnpstackZmeshgridr/ZastyperRarrayrr!Z from_numpyr )rFr3r-rbox_coordinatesr'r'r(!normalize_grid_corner_coordinates5s"  "z:OwlViTForObjectDetection.normalize_grid_corner_coordinates)r3r+cCs||}t|dd}t|dt| d}t|d|jd}t|dt| d}tj||gdd}|S)Nrrg-C6?rrvr)r9r!Zcliploglog1pZ full_likerr)rFr3r8Zbox_coord_biasZbox_sizeZ box_size_biasZbox_biasr'r'r(compute_box_biasJs z)OwlViTForObjectDetection.compute_box_bias) image_featsr3r+cCs&||}|||7}||}|S)a Args: image_feats: Features extracted from the image, returned by the `image_text_embedder` method. feature_map: A spatial re-arrangement of image_features, also returned by the `image_text_embedder` method. Returns: pred_boxes: List of predicted boxes (cxcywh normalized to 0, 1) nested within a dictionary. )r0r<r2)rFr=r3rjr'r'r( box_predictorZs  z&OwlViTForObjectDetection.box_predictorN)r=r'r(r+cCs||||\}}||fS)a8 Args: image_feats: Features extracted from the `image_text_embedder`. query_embeds: Text query embeddings. query_mask: Must be provided with query_embeddings. A mask indicating which query embeddings are valid. )r/)rFr=r'r(r-r,r'r'r(class_predictorqsz(OwlViTForObjectDetection.class_predictor)rrrrrr+c Cs|j|||||dd}|jd}|jj|}tt|jtd} t |ddddddf| } |ddddddf| }| |}|jdt t |jdt t |jd|jdf} | | }|d} | ||fS)NT)rrrrrrrrrrrrv)rr<rrrIr5r7rr! broadcast_tor1rUsqrtr) rFrrrrrrrr:new_sizeclass_token_outr9r'r'r(image_text_embedders,  "  z,OwlViTForObjectDetection.image_text_embedder)rrrr+c Cs|jj|dd}|d}|jj|}tt|jtd}t|ddddddf|}|ddddddf|}| |}|jdt t |jdt t |jd|jdf}| |}||fS)NT)rrrr@rrv) rrrrIr5r7rr!rBr1rUrCr) rFrrrr rr:rDrEr'r'r(image_embedders"  z'OwlViTForObjectDetection.image_embedder)query_image_featuresquery_feature_mapr+cCs>||\}}|||}t|}g}g}|j} t|jdD]} tjddddgg| d} || } t| | \} }t | ddkrt | | } t | d}| d|k }| r>|| |d}tj|| dd}td||}|t|}||| |||q>|r,t|}t|}nd\}}|||fS) Nrrr,rg?r4zd,id->i)NN)r?r>rr-rrr!rrbrcrgr[ZnonzeroZnumelZsqueezerr)Zargminappendr6)rFrHrIrrkrjZpred_boxes_as_cornersZbest_class_embedsbest_box_indicesZpred_boxes_deviceiZeach_query_boxZeach_query_pred_boxesZiousZ iou_thresholdZ selected_indsZselected_embeddingsZ mean_embedsZmean_simZ best_box_indr'Z box_indicesr'r'r(embed_image_querys4     z*OwlViTForObjectDetection.embed_image_queryr)rquery_pixel_valuesrrrr+c Cs"|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|j|dd}|j|||d\}}|j\} } } } t|| | | | f} |j\} } } } t|| | | | f} || |\}}}|j | |d\}}| | |}|s ||||||| f}t dd|D}|St ||||||d|dS) a3 Returns: Examples: ```python >>> import requests >>> from PIL import Image >>> import torch >>> from transformers import AutoProcessor, OwlViTForObjectDetection >>> processor = AutoProcessor.from_pretrained("google/owlvit-base-patch16") >>> model = OwlViTForObjectDetection.from_pretrained("google/owlvit-base-patch16") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> query_url = "http://images.cocodataset.org/val2017/000000001675.jpg" >>> query_image = Image.open(requests.get(query_url, stream=True).raw) >>> inputs = processor(images=image, query_images=query_image, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model.image_guided_detection(**inputs) >>> # Target image sizes (height, width) to rescale box predictions [batch_size, 2] >>> target_sizes = torch.Tensor([image.size[::-1]]) >>> # Convert outputs (bounding boxes and class logits) to COCO API >>> results = processor.post_process_image_guided_detection( ... outputs=outputs, threshold=0.6, nms_threshold=0.3, target_sizes=target_sizes ... ) >>> i = 0 # Retrieve predictions for the first image >>> boxes, scores = results[i]["boxes"], results[i]["scores"] >>> for box, score in zip(boxes, scores): ... box = [round(i, 2) for i in box.tolist()] ... print(f"Detected similar object with confidence {round(score.item(), 3)} at location {box}") Detected similar object with confidence 0.856 at location [10.94, 50.4, 315.8, 471.39] Detected similar object with confidence 1.0 at location [334.84, 25.33, 636.16, 374.71] ```N)rr)rrr)r=r'css|]}|dk r|VqdSrQr'rCxr'r'r(rGHszBOwlViTForObjectDetection.image_guided_detection..)r:rmrnror*rkr;r<)rrrrrrGrr!rrMr?r>rArIrl)rFrrNrrrrIr3r rr hidden_dimr=Zquery_image_featsr'rKror-rkrnrr'r'r(image_guided_detectionsL+   z/OwlViTForObjectDetection.image_guided_detection)rrrrrrr+c Cs2|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|j|||||d\}}} | j} | j} |j\} } } }t || | | |f}|jd| }| | ||jd}| | ||jd}|ddk}| |||\}}| ||}|s|||||| | f}t dd|D}|St|||||| | dS) a Returns: Examples: ```python >>> import requests >>> from PIL import Image >>> import torch >>> from transformers import AutoProcessor, OwlViTForObjectDetection >>> processor = AutoProcessor.from_pretrained("google/owlvit-base-patch32") >>> model = OwlViTForObjectDetection.from_pretrained("google/owlvit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> texts = [["a photo of a cat", "a photo of a dog"]] >>> inputs = processor(text=texts, images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> # Target image sizes (height, width) to rescale box predictions [batch_size, 2] >>> target_sizes = torch.Tensor([image.size[::-1]]) >>> # Convert outputs (bounding boxes and class logits) to final bounding boxes and scores >>> results = processor.post_process_object_detection( ... outputs=outputs, threshold=0.1, target_sizes=target_sizes ... ) >>> i = 0 # Retrieve predictions for the first image for the corresponding text queries >>> text = texts[i] >>> boxes, scores, labels = results[i]["boxes"], results[i]["scores"], results[i]["labels"] >>> for box, score, label in zip(boxes, scores, labels): ... box = [round(i, 2) for i in box.tolist()] ... print(f"Detected {text[label]} with confidence {round(score.item(), 3)} at location {box}") Detected a photo of a cat with confidence 0.707 at location [324.97, 20.44, 640.58, 373.29] Detected a photo of a cat with confidence 0.717 at location [1.46, 55.26, 315.55, 472.17] ```N)rrrrrrrv).rcss|]}|dk r|VqdSrQr'rOr'r'r(rGsz3OwlViTForObjectDetection.forward..)r:r9rjr*rkr;r<)rrrrrrFr;r<rr!rr?r>rArIrh)rFrrrrrrr'r3rrr rrrQr=Zmax_text_queriesr(r-rkrjrr'r'r(rVsR/    z OwlViTForObjectDetection.forward)NN)NN)NN)NNNN)NNNN)rKrLrMrrrzr!rOr9r<r>rrrr?r"rFrGrMr5OWLVIT_IMAGE_GUIDED_OBJECT_DETECTION_INPUTS_DOCSTRINGrrlrR(OWLVIT_OBJECT_DETECTION_INPUTS_DOCSTRINGrhrrr'r'rr(r.(s    - ! ) ] r.)N)r)NrNr dataclassesrtypingrrrrrnumpyr5r!Ztorch.utils.checkpointrr Z activationsr Zmodeling_outputsr r Zmodeling_utilsrrrrrrrrZconfiguration_owlvitrrrZtransformers.image_transformsrZ get_loggerrKloggerZ_CHECKPOINT_FOR_DOCZ$OWLVIT_PRETRAINED_MODEL_ARCHIVE_LISTrrUr)r1r4r5rVrYrbrgrhrlrrprrrrrZOWLVIT_START_DOCSTRINGrrrrTrSrSizer-rrrrrrrr#r.r'r'r'r(s~      & 10 l2: a H4:2W1