U ,-e@sdZddlZddlZddlmZddlmZmZmZm Z ddl Z ddl Z ddl m Z ddl mZddlmZmZdd lmZdd lmZmZmZmZmZd d lmZmZmZeeZ d Z!d gZ"dCe j#e j$ee%dddZ&e j#e j#dddZ'e j#e j#dddZ(eGdddeZ)eGdddeZ*eGdddeZ+Gddde j,Z-Gdd d e j,Z.Gd!d"d"e j,Z/Gd#d$d$e j,Z0Gd%d&d&e j,Z1Gd'd(d(eZ2d)Z3d*Z4d+Z5d,Z6Gd-d.d.e j,Z7dDe j8e j$e j9e%d/d0d1Z:Gd2d3d3e j,Z;Gd4d5d5e2Zee3Gd:d;d;e2Z?Gdd?d?e2ZAed@e3GdAdBdBe2ZBdS)Ez PyTorch CLIPSeg model.N) dataclass)AnyOptionalTupleUnion)nn)ACT2FN)BaseModelOutputBaseModelOutputWithPooling)PreTrainedModel) ModelOutputadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) CLIPSegConfigCLIPSegTextConfigCLIPSegVisionConfigzCIDAS/clipseg-rd64-refined)maskdtypetgt_lencCsj|\}}|dk r|n|}|ddddddf|d|||}d|}||tjt|jS)z_ Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`. Nr?)sizeexpandtoZ masked_filltorchboolfinfomin)rrrbszsrc_lenZ expanded_maskZ inverted_maskr#m/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/models/clipseg/modeling_clipseg.py _expand_mask3s  *r%)logitsreturncCstj|tjt||jdS)Ndevice)r functionalZ cross_entropyrarangelenr))r&r#r#r$contrastive_lossCsr-) similarityr'cCs t|}t|}||dS)Ng@)r-t)r.Z caption_lossZ image_lossr#r#r$ clipseg_lossHs r0c@seZdZUdZdZeejed<dZ ejed<dZ ejed<dZ ejed<dZ ejed<dZ eed<dZeed <eed d d ZdS) CLIPSegOutputa Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image:(`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text:(`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds(`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`CLIPSegTextModel`]. image_embeds(`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`CLIPSegVisionModel`]. text_model_output(`BaseModelOutputWithPooling`): The output of the [`CLIPSegTextModel`]. vision_model_output(`BaseModelOutputWithPooling`): The output of the [`CLIPSegVisionModel`]. Nlosslogits_per_imagelogits_per_text text_embeds image_embedstext_model_outputvision_model_outputr'cstfddDS)Nc3s,|]$}|dkr|n t|VqdS))r7r8Ngetattrto_tuple.0kselfr#r$ osz)CLIPSegOutput.to_tuple..tuplekeysr@r#r@r$r<ns zCLIPSegOutput.to_tuple)__name__ __module__ __qualname____doc__r2rr FloatTensor__annotations__r3r4r5r6r7r r8rrr<r#r#r#r$r1Ns   r1c@sLeZdZUdZdZejed<dZe e ejed<dZ e e ejed<dS)CLIPSegDecoderOutputa Args: logits (`torch.FloatTensor` of shape `(batch_size, height, width)`): Classification scores for each pixel. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nr& hidden_states attentions) rFrGrHrIr&rrJrKrMrrrNr#r#r#r$rLus  rLc@sxeZdZUdZdZeejed<dZ ejed<dZ ejed<dZ ejed<dZ e ed<dZeed<eed d d ZdS) CLIPSegImageSegmentationOutputa, Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. ... vision_model_output (`BaseModelOutputWithPooling`): The output of the [`CLIPSegVisionModel`]. Nr2r&conditional_embeddings pooled_outputr8decoder_outputr9cstfddDS)Nc3s,|]$}|dkr|n t|VqdS))r8rRNr:r=r@r#r$rBsz:CLIPSegImageSegmentationOutput.to_tuple..rCr@r#r@r$r<s z'CLIPSegImageSegmentationOutput.to_tuple)rFrGrHrIr2rrrJrKr&rPrQr8r rRrLrrr<r#r#r#r$rOs    rOcs>eZdZedfdd ZddZejejdddZ Z S) CLIPSegVisionEmbeddingsconfigcst||_|j|_|j|_|j|_tt |j|_ tj |j |j|j|jdd|_|j|jd|_|jd|_t|j|j|_|jdt |jddddS)NF)Z in_channelsZ out_channels kernel_sizestridebiasr position_idsr persistent)super__init__rU hidden_size embed_dimZ image_size patch_sizer ParameterrZrandnclass_embeddingConv2dZ num_channelspatch_embedding num_patches num_positions Embeddingposition_embeddingregister_bufferr+rrArU __class__r#r$r`s"  z CLIPSegVisionEmbeddings.__init__cCst|dkrtdt|jd}|jjddjd|jj ||}t j j ||ddd d|jj |d|dj}t|jjdd|g}|S) NrYz#new_size should consist of 2 valuesg?rZbicubicF)modeZ align_cornersr)r, ValueErrorintrhrkweightTviewrUrarr*Z interpolatesqueezercat)rAnew_sizeZnum_patches_one_directionabresultr#r#r$interpolate_position_embeddingss$ z7CLIPSegVisionEmbeddings.interpolate_position_embeddings) pixel_valuesr'cCs|jd}||}|ddd}|j|dd}tj||gdd}|jd|jkrt t |jdd}|| ||f}| |j}n|||j}|S)NrrYrr\dim)shapergflatten transposererrrwrirrmathsqrtr|rrrkrZ)rAr} batch_sizeZ patch_embedsZ class_embeds embeddingsZ new_shaper#r#r$forwards  zCLIPSegVisionEmbeddings.forward) rFrGrHrr`r|rrJTensorr __classcell__r#r#rnr$rSsrScsLeZdZedfdd Zdeejeejeejej dddZ Z S) CLIPSegTextEmbeddingsrTcsRt|j}t|j||_t|j||_|j dt |j ddddS)NrZr[Fr]) r_r`rarrjZ vocab_sizetoken_embeddingZmax_position_embeddingsrkrlrr+rrArUrbrnr#r$r`s zCLIPSegTextEmbeddings.__init__N) input_idsrZ inputs_embedsr'cCsb|dk r|jdn|jd}|dkr:|jddd|f}|dkrL||}||}||}|S)Nr\)rrZrrk)rArrZrZ seq_lengthZposition_embeddingsrr#r#r$rs  zCLIPSegTextEmbeddings.forward)NNN) rFrGrHrr`rr LongTensorrJrrrr#r#rnr$rsrc szeZdZdZfddZejeedddZd eje eje eje e e eje eje e ejfd d d Z Z S) CLIPSegAttentionz=Multi-headed attention from 'Attention Is All You Need' papercst||_|j|_|j|_|j|j|_|j|j|jkrZtd|jd|jd|jd|_ |j |_ t |j|j|_t |j|j|_t |j|j|_t |j|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).)r_r`rUrarbnum_attention_heads num_headshead_dimrqscaleZattention_dropoutdropoutrLineark_projv_projq_projout_projrmrnr#r$r`s  zCLIPSegAttention.__init__)tensorseq_lenr!cCs ||||j|jddS)NrrY)rurrr contiguous)rArrr!r#r#r$_shapeszCLIPSegAttention._shapeNFrMattention_maskcausal_attention_maskoutput_attentionsr'cCs|\}}}|||j}|||d|} |||d|} ||jd|jf} ||||j| }| j| } | j| } | d} t || dd} | ||j|| fkrt d||j|| fd| |dk rD||d|| fkrt d|d|| fd|| ||j|| |} | ||j|| } |dk r||d|| fkrt d|d|| fd|| ||j|| |} | ||j|| } t jj| dd} |r| ||j|| }|||j|| } nd}t jj| |j|jd }t || }|||j||jfkrRt d ||j||jfd||||j||j}| dd}||||}||}||fS) z#Input shape: Batch x Time x Channelr\rrYz$Attention weights should be of size z , but is Nz!Attention mask should be of size r~)ptrainingz `attn_output` should be of size )rrrrrrrrrurZbmmrrqrr*ZsoftmaxrrZreshaper)rArMrrrr!rrbZ query_statesZ key_statesZ value_statesZ proj_shaper" attn_weightsZattn_weights_reshapedZ attn_probsZ attn_outputr#r#r$rsX         zCLIPSegAttention.forward)NNF)rFrGrHrIr`rrrrrrrrrrr#r#rnr$rs rcs0eZdZfddZejejdddZZS) CLIPSegMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSN) r_r`rUr hidden_act activation_fnrrraintermediate_sizefc1fc2rmrnr#r$r`ls   zCLIPSegMLP.__init__)rMr'cCs"||}||}||}|Sr)rrr)rArMr#r#r$rss   zCLIPSegMLP.forward)rFrGrHr`rrrrr#r#rnr$rks rcsJeZdZedfdd Zdejejejeee ej dddZ Z S) CLIPSegEncoderLayerrTcsRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSN)Zeps r_r`rarbr self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2rmrnr#r$r`|s    zCLIPSegEncoderLayer.__init__FrcCsd|}||}|j||||d\}}||}|}||}||}||}|f}|r`||f7}|SaI Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. `(config.encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rMrrr)rrrrrArMrrrZresidualroutputsr#r#r$rs"     zCLIPSegEncoderLayer.forward)F) rFrGrHrr`rrrrrrJrrr#r#rnr$r{s rc@s.eZdZdZeZdZdZddZd ddZ d S) CLIPSegPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. clipTcCsX|jj}t|trF|jjjjd|dd|jjjjd|ddnt|t r|jj}t j j|j d|j d|dt j j|jj|jj|dt j j|jj|jj|dnTt|trD|jj}|j dd|jjd|}|j d|}t j j|jj|dt j j|jj|dt j j|jj|dt j j|jj|dnt|tr|jj}|jjdd|jjd|}d|jjd|}t j j|jj|dt j j|jj|dnPt|trt j j|jj|jd|jjdt j j|jj|jd|jjdt|t jr.|j j!|jj"dt|t j#rT|j dk rT|j j!dS) zInitialize the weightsgg{Gz?)Zmeanstdr)rrYrN)$rUZinitializer_factor isinstancerrrsdataZnormal_rkrSrinitrerbrgZinitializer_rangernum_hidden_layersrrrrrrarr CLIPSegModeltext_projectiontext_embed_dimvisual_projectionvision_embed_dimrrXZzero_Zfill_r)rAmodulefactorZ in_proj_stdZ out_proj_stdZfc_stdr#r#r$ _init_weightssL       z$CLIPSegPreTrainedModel._init_weightsFcCst|tr||_dSr)rCLIPSegEncodergradient_checkpointing)rArvaluer#r#r$_set_gradient_checkpointings z2CLIPSegPreTrainedModel._set_gradient_checkpointingN)F) rFrGrHrIr config_classZbase_model_prefixZsupports_gradient_checkpointingrrr#r#r#r$rs +raI This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`CLIPSegConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. aE Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using [`AutoImageProcessor`]. See [`CLIPImageProcessor.__call__`] for details. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using [`AutoImageProcessor`]. See [`CLIPImageProcessor.__call__`] for details. return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. c s`eZdZdZedfdd Zd eejeejee ee ee e e e fdddZ ZS) rz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`CLIPSegEncoderLayer`]. Args: config: CLIPSegConfig rTcs:t|_tfddtjD|_d|_dS)Ncsg|] }tqSr#)rr>_rTr#r$ Tsz+CLIPSegEncoder.__init__..F) r_r`rUr ModuleListrangerlayersrrmrnrTr$r`Qs  zCLIPSegEncoder.__init__N)rrroutput_hidden_states return_dictr'csdk r n|jj|dk r |n|jj}|dk r4|n|jj}|rDdnd}rPdnd}|} t|jD]r\} } |rx|| f}|jr|jrfdd} tj j | | | ||} n| | ||d} | d} rb|| df}qb|r|| f}|st dd | ||fDSt | ||d S) a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr#csfdd}|S)Ncs|fSrr#)inputs)rrr#r$custom_forwardszMCLIPSegEncoder.forward..create_custom_forward..custom_forwardr#)rrr)rr$create_custom_forwardsz5CLIPSegEncoder.forward..create_custom_forwardrrrcss|]}|dk r|VqdSrr#r>vr#r#r$rBsz)CLIPSegEncoder.forward..)last_hidden_staterMrN) rUrruse_return_dict enumeraterrrrutils checkpointrDr )rArrrrrrZencoder_statesall_attentionsrMidxZ encoder_layerr layer_outputsr#rr$rWsH&      zCLIPSegEncoder.forward)NNNNN)rFrGrHrIrr`rrrrrrr rrr#r#rnr$rHs  r)input_ids_shaperr)past_key_values_lengthcCs|\}}tj||ft|j|d}tj|d|d}|||d|ddkd||}|dkrtj tj ||||d|gdd}|ddddddf |d|||S)zB Make causal mask used for bi-directional self-attention. r(r\rrrr)r~N) rfullrr r+rZ masked_fill_rurrwZzerosr)rrr)rr!rrZ mask_condr#r#r$_make_causal_masks"  rc sxeZdZedfdd Zeeeeedd e e j e e j e e j e e e e e e e eefdddZZS) CLIPSegTextTransformerrTcsHt||_|j}t||_t||_tj ||j d|_ |j |_ dSr) r_r`rUrarrrencoderrrrfinal_layer_norm eos_token_idrrnr#r$r`s   zCLIPSegTextTransformer.__init__ output_typerNrrrZrrrr'c Csn|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|dkrLtd|}|d|d}|j||d}t||j |j d} |dk rt ||j }|j ||| |||d} | d} | | } |jdkr| tj| jd| j d|jtj| j d jdd f} n>| tj| jd| j d|jtj| j d |jkjdd f} |sZ| | f| d dSt| | | j| jd S)  Returns: NzYou have to specify input_idsr\)rrZr()rrrrrrrrYrr~rr pooler_outputrMrN)rUrrrrqrrurrrr)r%rrrrr+rrrrZargmaxr rMrN) rArrrZrrrZ input_shaperMrencoder_outputsrrQr#r#r$rsV    zCLIPSegTextTransformer.forward)NNNNNN)rFrGrHrr`rCLIPSEG_TEXT_INPUTS_DOCSTRINGrr rrrrrrrrr#r#rnr$rs$   rc seZdZeZddgZedfdd ZejdddZ d d Z e e e eed deejeejeejeeeeeeeeefd ddZZS)CLIPSegTextModelrrrTcs"t|t||_|dSr)r_r`r text_model post_initrmrnr#r$r`%s  zCLIPSegTextModel.__init__r9cCs |jjjSrrrrr@r#r#r$get_input_embeddings+sz%CLIPSegTextModel.get_input_embeddingscCs||jj_dSrr)rArr#r#r$set_input_embeddings.sz%CLIPSegTextModel.set_input_embeddingsrNrcCs|j||||||dS)aM Returns: Examples: ```python >>> from transformers import AutoTokenizer, CLIPSegTextModel >>> tokenizer = AutoTokenizer.from_pretrained("CIDAS/clipseg-rd64-refined") >>> model = CLIPSegTextModel.from_pretrained("CIDAS/clipseg-rd64-refined") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled (EOS token) states ```rrrZrrr)r)rArrrZrrrr#r#r$r1szCLIPSegTextModel.forward)NNNNNN)rFrGrHrrZ_no_split_modulesr`rModulerrrrrr rrrrrrrrr#r#rnr$r s,  rc sheZdZedfdd Zeeeeedd e e j e e e e e e e eefdddZZS) CLIPSegVisionTransformerrTcsRt||_|j}t||_tj||jd|_ t ||_ tj||jd|_ dSr) r_r`rUrarSrrrr pre_layrnormrrpost_layernormrrnr#r$r`Ys   z!CLIPSegVisionTransformer.__init__rNr}rrrr'c Cs|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|dkrLtd||}||}|j||||d}|d}|dddddf}||}|s||f|ddSt |||j |j dS)rNz You have to specify pixel_values)rrrrrrr) rUrrrrqrrrrr rMrN) rAr}rrrrMrrrQr#r#r$rcs2   z CLIPSegVisionTransformer.forward)NNNN)rFrGrHrr`rCLIPSEG_VISION_INPUTS_DOCSTRINGrr rrrJrrrrrr#r#rnr$rWs   rc seZdZeZdZedfdd ZejdddZ e e e e edd eejeeeeeeeee fd d d ZZS)CLIPSegVisionModelr}rTcs"t|t||_|dSr)r_r`r vision_modelrrmrnr#r$r`s  zCLIPSegVisionModel.__init__r9cCs |jjjSr)rrrgr@r#r#r$rsz'CLIPSegVisionModel.get_input_embeddingsrNrcCs|j||||dS)a Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, CLIPSegVisionModel >>> processor = AutoProcessor.from_pretrained("CIDAS/clipseg-rd64-refined") >>> model = CLIPSegVisionModel.from_pretrained("CIDAS/clipseg-rd64-refined") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled CLS states ```r}rrr)r)rAr}rrrr#r#r$rs zCLIPSegVisionModel.forward)NNNN)rFrGrHrrZmain_input_namer`rrrrrrr rrrJrrrrrr#r#rnr$rs"  rcseZdZeZedfdd Zeedee j ee j ee j ee ee ee e j dddZ eedee j ee ee ee e j dd d Zeeeeed dee jee j ee j ee jee ee ee ee eeefd d dZZS)rrTcst|t|jts.tdt|jdt|jtsPtdt|jd|j}|j}|j |_ |j |_ |j |_ t ||_t||_tj|j |j dd|_tj|j |j dd|_tt|jj|_|dS)NzNconfig.text_config is expected to be of type CLIPSegTextConfig but is of type .zRconfig.vision_config is expected to be of type CLIPSegVisionConfig but is of type F)rX)r_r`r text_configrrqtype vision_configrprojection_dimrarrrrrrrrrrrdrrrUZlogit_scale_init_value logit_scaler)rArUrr rnr#r$r`s(     zCLIPSegModel.__init__Nrc Csh|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|j||||||d}|d}||} | S)a Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`CLIPSegTextModel`]. Examples: ```python >>> from transformers import AutoTokenizer, CLIPSegModel >>> tokenizer = AutoTokenizer.from_pretrained("CIDAS/clipseg-rd64-refined") >>> model = CLIPSegModel.from_pretrained("CIDAS/clipseg-rd64-refined") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> text_features = model.get_text_features(**inputs) ```Nrr)rUrrrrr) rArrrZrrr text_outputsrQZ text_featuresr#r#r$get_text_featuress  zCLIPSegModel.get_text_featuresrcCsd|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|j||||d}|d}||}|S)aI Returns: image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`CLIPSegVisionModel`]. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, CLIPSegModel >>> processor = AutoProcessor.from_pretrained("CIDAS/clipseg-rd64-refined") >>> model = CLIPSegModel.from_pretrained("CIDAS/clipseg-rd64-refined") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> image_features = model.get_image_features(**inputs) ```Nrr)rUrrrrr)rAr}rrrvision_outputsrQZimage_featuresr#r#r$get_image_featuress zCLIPSegModel.get_image_featuresr) rr}rrZ return_lossrrrr'c  Cs*|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|j||||d} |j||||||d} | d} || } | d} || } | | jdddd} | | jdddd} |j } t | | | }| }d}|rt|}|s||| | | | f}|dk r|f|S|St|||| | | | d S) a Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, CLIPSegModel >>> processor = AutoProcessor.from_pretrained("CIDAS/clipseg-rd64-refined") >>> model = CLIPSegModel.from_pretrained("CIDAS/clipseg-rd64-refined") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor( ... text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True ... ) >>> outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities ```NrrrrYr\T)rrZkeepdim)r2r3r4r5r6r7r8)rUrrrrrrrZnormr exprmatmulr/r0r1)rArr}rrZrrrrrr r6r5r r4r3r2outputr#r#r$rKsT&    zCLIPSegModel.forward)NNNNNN)NNNN)NNNNNNNN)rFrGrHrrr`rrrrrrrJr rrCLIPSEG_INPUTS_DOCSTRINGrr1rrrrrr#r#rnr$rsf .0  rcsNeZdZdZedfdd Zd ejejejee e ej dddZ Z S) CLIPSegDecoderLayerz CLIPSeg decoder layer, which is identical to `CLIPSegEncoderLayer`, except that normalization is applied after self-attention/MLP, rather than before. rTcsRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSrrrmrnr#r$r`s    zCLIPSegDecoderLayer.__init__FrcCsd|}|j||||d\}}||}||}|}||}||}||}|f}|r`||f7}|Sr)rrrrrr#r#r$rs"     zCLIPSegDecoderLayer.forward)F)rFrGrHrIrr`rrrrrrJrrr#r#rnr$rs rcsNeZdZedfdd Zd eejejee ee ee dddZ Z S) CLIPSegDecoderrTc sXtj|_tjj|_tjj|_j rj j dj j df}t tj jjdddttjjjd|d|ddttjjdd|d|dd|_ntjjdj j j j d|_tj}tfd d t|D|_tj j_j_j_d _tfd d ttjD|_dS) Nrr)rVpaddingrYr)rVrW)rWcsg|]}tjjjqSr#)rrr ra reduce_dimrrTr#r$rsz+CLIPSegDecoder.__init__..Zrelucsg|] }tqSr#)rr)decoder_configr#r$r s)r_r`conditional_layerrrr rfilm_mulfilm_addZ"use_complex_transposed_convolutionr rcZ SequentialrfZReLUZConvTranspose2dtransposed_convolutionr,extract_layersrrreducescopydeepcopyraZdecoder_num_attention_headsrZdecoder_intermediate_sizerrr)rArUZtransposed_kernelsdepthrn)rUrr$r`sN   zCLIPSegDecoder.__init__NT)rMrPrrrcCsp|rdnd}|rdnd}|ddd}d} tt||j|jD]\} \} } } | dk rb| | | } n| | } | |jkr||| ddd||} | ddd} | | dd|d}|d} |r|| f7}|r>||df7}q>| ddddddfddd} tt | j d}|j d}| || j d||} | | }|sbtdd|||fDSt|||d S) Nr#r\rrrY)rrrcss|]}|dk r|VqdSrr#rr#r#r$rB=sz)CLIPSegDecoder.forward..)r&rMrN)rziprr rrZpermuterrrrrrrurrvrDrL)rArMrPrrrZall_hidden_statesr activationsriZ activationlayerreducerrrr&r#r#r$r sH  "  $ zCLIPSegDecoder.forward)NNT) rFrGrHrr`rrrrrrrr#r#rnr$rs.rzn CLIPSeg model with a Transformer-based decoder on top for zero-shot and one-shot image segmentation. cseZdZeZedfdd Zd eeej eej eej eej dddZ e e e eedd eejeejeejeejeej eejeejeeeeeeeeefd d d ZZS)CLIPSegForImageSegmentationrTcs:t|||_t||_|j|_t||_|dSr) r_r`rUrrrrdecoderrrmrnr#r$r`Os    z$CLIPSegForImageSegmentation.__init__NrrrrZconditional_pixel_valuesc Cs|dk rDt||krtdt|jj|||d}W5QRXnF|dk rt||kr`tdt|j|}W5QRXntd|S)Nz@Make sure to pass as many prompt texts as there are query images)rrZzAMake sure to pass as many prompt images as there are query imagesz[Invalid conditional, should be either provided as `input_ids` or `conditional_pixel_values`)r,rqrno_gradrr r)rArrrrZr,rPr#r#r$get_conditional_embeddings\s$    z6CLIPSegForImageSegmentation.get_conditional_embeddingsr) rr}r,rPrrZlabelsrrrr'c  s| dk r | n|jj} t|jj||d| d} |j| d} | rL| jn| dfdd|jD} | rt | j | j | r| jnd| j d} n | s| dd| d dn| } W5QRX|dkr|j |jd ||||d }n8|jd |jd krtd |jd|jjkrtd |j| ||| | d}| r4|jn|d }d}|dk rh||j}t}|||}| s||| | |f}|dk r|f|S|St|||| | |dS)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Returns: Examples: ```python >>> from transformers import AutoProcessor, CLIPSegForImageSegmentation >>> from PIL import Image >>> import requests >>> processor = AutoProcessor.from_pretrained("CIDAS/clipseg-rd64-refined") >>> model = CLIPSegForImageSegmentation.from_pretrained("CIDAS/clipseg-rd64-refined") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> texts = ["a cat", "a remote", "a blanket"] >>> inputs = processor(text=texts, images=[image] * len(texts), padding=True, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> print(logits.shape) torch.Size([3, 352, 352]) ```NTrrrYcsg|]}|dqS)rr#)r>r&rMr#r$rsz7CLIPSegForImageSegmentation.forward..rrrr+zWMake sure to pass as many conditional embeddings as there are query images in the batchzcMake sure that the feature dimension of the conditional embeddings matches `config.projection_dim`.)rrr)r2r&rPrQr8rR)rUrrr-rrrrMrr rrrNr.rrqr r*r&rr)rZBCEWithLogitsLossrO)rArr}r,rPrrZr/rrrrrQr%Zdecoder_outputsr&r2Zloss_fnrr#r0r$rysx,      z#CLIPSegForImageSegmentation.forward)NNNNN) NNNNNNNNNN)rFrGrHrrr`rrrrrr.rrrrOrrJrrrrr1rrr#r#rnr$r)FsN   r))N)r)CrIr!r dataclassesrtypingrrrrrZtorch.utils.checkpointrr%r Zmodeling_outputsr r Zmodeling_utilsr rr rrrrZconfiguration_clipsegrrrZ get_loggerrFloggerZ_CHECKPOINT_FOR_DOCZ%CLIPSEG_PRETRAINED_MODEL_ARCHIVE_LISTrrrrr%r-r0r1rLrOrrSrrrrrZCLIPSEG_START_DOCSTRINGrrrrSizer)rrrrrrrrr)r#r#r#r$sr     %>"i2:  'i _7<4b9d