U 9%e@s`dZddlZddlmZddlmZmZmZmZddl Z ddl Z ddl m Z ddl m Z ddlmZdd lmZmZdd lmZdd lmZmZmZmZmZd d lmZmZmZd dlm Z m!Z!e"e#Z$dZ%ddddddddgZ&e j'e j'dddZ(e j'e j'dddZ)eGdddeZ*eGdd d eZ+eGd!d"d"eZ,eGd#d$d$eZ-Gd%d&d&e j.Z/Gd'd(d(e j.Z0Gd)d*d*e j.Z1Gd+d,d,e j.Z2Gd-d.d.e j.Z3Gd/d0d0eZ4d1Z5d2Z6d3Z7d4Z8Gd5d6d6e j.Z9Gd7d8d8e4Z:ee5Gd9d:d:e4Z;ed;e5Gde5Gd?d@d@e4Z=edAe5GdBdCdCe4Z>dS)Dz PyTorch BLIP model.N) dataclass)AnyOptionalTupleUnion)nn) normalize)ACT2FN)BaseModelOutputBaseModelOutputWithPooling)PreTrainedModel) ModelOutputadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) BlipConfigBlipTextConfigBlipVisionConfig)BlipTextLMHeadModel BlipTextModelzSalesforce/blip-vqa-basez!Salesforce/blip-vqa-capfilt-largez%Salesforce/blip-image-captioning-basez&Salesforce/blip-image-captioning-largezSalesforce/blip-itm-base-cocozSalesforce/blip-itm-large-cocozSalesforce/blip-itm-base-flickrz Salesforce/blip-itm-large-flickr)logitsreturncCstj|tjt||jdS)N)device)r functionalZ cross_entropytorcharangelenr)rr e/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/transformers/models/blip/modeling_blip.pycontrastive_loss:sr") similarityrcCs t|}t|}||dS)Ng@)r"t)r#Z caption_lossZ image_lossr r r! blip_loss?s r%c@seZdZUdZdZeeeje d<dZ eeeje d<dZ eeje d<dZ eje d<dZ eeeje d<dZeeeje d<ed d ZdS) 'BlipForConditionalGenerationModelOutputa Adapted from the base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. This class also adds the loss term from the text decoder. Args: loss (`torch.FloatTensor`, *optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Languge modeling loss from the text decoder. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`, *optional*): Prediction scores of the language modeling head of the text decoder model. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*): The image embeddings obtained after applying the Vision Transformer model to the input image. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlossr image_embedslast_hidden_state hidden_states attentionscCstdt|jS)Nz`decoder_logits` attribute is deprecated and will be removed in version 5 of Transformers. Please use the `logits` attribute to retrieve the final output instead.)warningswarn FutureWarningrselfr r r!decoder_logitshs z6BlipForConditionalGenerationModelOutput.decoder_logits)__name__ __module__ __qualname____doc__r'rrr FloatTensor__annotations__rr(r)r*r+propertyr1r r r r!r&Es r&c@speZdZUdZdZeejed<dZ eejed<dZ ejed<dZ ee ejed<dZ ee ejed<dS)BlipTextVisionModelOutputa Adapted from the base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. This class also adds the loss term from the text decoder. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Languge modeling loss from the text decoder. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nr'r(r)r*r+)r2r3r4r5r'rrr6r7r(r)r*rr+r r r r!r9rs r9c@seZdZUdZdZeejed<dZ eejed<dZ eejed<dZ ejed<dZ ee ejed<dZeejed<dZee ejed <dZee ejed <dS) BlipImageTextMatchingModelOutputa Adapted from the base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. This class also adds the loss term from the text decoder as well as the image-text similarity scores. Args: itm_score (`torch.FloatTensor`): The image-text similarity scores. loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Languge modeling loss from the text decoder. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. vision_pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`, *optional*): Last layer hidden-state of the vision of the vision-only branch of the model. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. question_embeds (`torch.FloatTensor`): The question embeddings obtained by the text projection layer. N itm_scorer'r(r)r*vision_pooler_outputr+question_embeds)r2r3r4r5r;rrr6r7r'r(r)r*rr<r+r=r r r r!r:s r:c@seZdZUdZdZeejed<dZ ejed<dZ ejed<dZ ejed<dZ ejed<dZ eed<dZeed <eed d d ZdS) BlipOutputa Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image:(`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text:(`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds(`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`BlipTextModel`]. image_embeds(`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`BlipVisionModel`]. text_model_output(`BaseModelOutputWithPooling`): The output of the [`BlipTextModel`]. vision_model_output(`BaseModelOutputWithPooling`): The output of the [`BlipVisionModel`]. Nr'logits_per_imagelogits_per_text text_embedsr(text_model_outputvision_model_outputrcstfddDS)Nc3s,|]$}|dkr|n t|VqdS))rBrCN)getattrto_tuple).0kr/r r! sz&BlipOutput.to_tuple..)tuplekeysr/r r/r!rFs zBlipOutput.to_tuple)r2r3r4r5r'rrr6r7r?r@rAr(rBr rCrrrFr r r r!r>s   r>cs6eZdZedfdd ZejejdddZZ S)BlipVisionEmbeddingsconfigcst||_|j|_|j|_|j|_tt dd|j|_ tj d|j|j|jd|_ |j|jd|_|jd|_tt d|j|j|_dS)Nrr )Z in_channelsZ out_channelsZ kernel_sizeZstride)super__init__rN hidden_size embed_dimZ image_sizeZ patch_sizer ParameterrZrandnclass_embeddingConv2dpatch_embeddingZ num_patchesZ num_positionsposition_embeddingr0rN __class__r r!rQs  zBlipVisionEmbeddings.__init__) pixel_valuesrcCs|jd}|jjj}||j|d}|ddd}|j|dd|}t j ||gdd}||j ddd| dddf|}|S)NrdtyperOrdim) shaperWweightr^toflatten transposerUexpandrcatrXsize)r0r\ batch_sizeZ target_dtypeZ patch_embedsZ class_embeds embeddingsr r r!forwards  ,zBlipVisionEmbeddings.forward) r2r3r4rrQrr6Tensorrl __classcell__r r rZr!rLsrLcsLeZdZedfdd Zdeejeejeejej dddZ Z S) BlipTextEmbeddingsrMcsRt|j}t|j||_t|j||_|j dt |j ddddS)N position_ids)rr_F) persistent) rPrQrRr EmbeddingZ vocab_sizetoken_embeddingZmax_position_embeddingsrXZregister_bufferrrrgr0rNrSrZr r!rQs zBlipTextEmbeddings.__init__N) input_idsrp inputs_embedsrcCsb|dk r|jdn|jd}|dkr:|jddd|f}|dkrL||}||}||}|S)Nr_)rbrprsrX)r0rurprvZ seq_lengthZposition_embeddingsrkr r r!rls  zBlipTextEmbeddings.forward)NNN) r2r3r4rrQrr LongTensorr6rmrlrnr r rZr!rosroc sreZdZdZfddZejeedddZd eje eje e e eje eje e ejfd d d Z Z S) BlipAttentionz=Multi-headed attention from 'Attention Is All You Need' papercst||_|j|_|j|_|j|j|_|j|j|jkrZtd|jd|jd|jd|_ t |j |_ t |jd|j|_t |j|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gr )rPrQrNrRrSZnum_attention_heads num_headshead_dim ValueErrorscalerZDropoutZattention_dropoutdropoutLinearqkv projectionrYrZr r!rQ's  zBlipAttention.__init__)tensorseq_lenbszcCs ||||j|jddS)NrrO)viewrzr{rf contiguous)r0rrrr r r!_shape9szBlipAttention._shapeNF)r* head_maskoutput_attentionsrcCs|\}}}||||d|j||jddddd}|d|d|d}} } t|| dd} | |j} t j j | dd} | | } |d k r| |} t| | dddd} | d d|j f}| |} || }|r|| fn|d f}|S) z#Input shape: Batch x Time x Channelr rOrrr_rwr`N)rirZreshaperzZpermutermatmulrfr}rrZsoftmaxr~rSr)r0r*rrrZtgt_lenrSZ mixed_qkvZ query_statesZ key_statesZ value_statesZattention_scoresZattention_probsZ context_layerZnew_context_layer_shapeoutputoutputsr r r!rl<s8     zBlipAttention.forward)NF)r2r3r4r5rQrrmintrrboolrrlrnr r rZr!ry$s rycs0eZdZfddZejejdddZZS)BlipMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSN) rPrQrNr Z hidden_act activation_fnrrrRZintermediate_sizefc1fc2rYrZr r!rQks   zBlipMLP.__init__)r*rcCs"||}||}||}|Sr)rrr)r0r*r r r!rlrs   zBlipMLP.forward)r2r3r4rQrrmrlrnr r rZr!rjs rcsFeZdZedfdd Zdejejeee ej dddZ Z S) BlipEncoderLayerrMcsRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSN)Zeps) rPrQrRrSry self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2rYrZr r!rQzs    zBlipEncoderLayer.__init__F)r*attention_maskrrcCsb|}||}|j|||d\}}||}|}||}||}||}|f}|r^||f7}|S)aI Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. `(config.encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )r*rr)rrrr)r0r*rrZresidualZ attn_weightsrr r r!rls      zBlipEncoderLayer.forward)F) r2r3r4rrQrrmrrrr6rlrnr r rZr!rys rc@s.eZdZdZeZdZdZddZd ddZ d S) BlipPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. ZblipTcCs|jj}t|tjs,t|tjs,t|tjr^|jjj d|dt |dr^|j dk r^|j j t|t rt |jdr~|jjj}tjj|jd|dtjj|jd|dnJt|tjr|j j |jjdn"t|tjr|j dk r|j j dS)zInitialize the weightsg)meanZstdbiasN vision_configg?)rNZinitializer_range isinstancerrVrrrrcdataZnormal_hasattrrZzero_rLrinitZ trunc_normal_rXrUrZfill_)r0modulefactorr r r! _init_weightss.$      z!BlipPreTrainedModel._init_weightsFcCst|tr||_dSr)r BlipEncodergradient_checkpointing)r0rvaluer r r!_set_gradient_checkpointings z/BlipPreTrainedModel._set_gradient_checkpointingN)F) r2r3r4r5r config_classZbase_model_prefixZsupports_gradient_checkpointingrrr r r r!rs ra= This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`BlipConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. aN Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoProcessor`]. See [`BlipProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. aE Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using [`BlipImageProcessor`]. See [`BlipImageProcessor.__call__`] for details. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoProcessor`]. See [`BlipProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using [`BlipImageProcessor`]. See [`BlipImageProcessor.__call__`] for details. return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. csXeZdZdZedfdd Zd eejee ee ee e e e fdddZ ZS) rz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`BlipEncoderLayer`]. Args: config (`BlipConfig`): The corresponding vision configuration for the `BlipEncoder`. rMcs:t|_tfddtjD|_d|_dS)Ncsg|] }tqSr )r)rG_rMr r! Dsz(BlipEncoder.__init__..F) rPrQrNrZ ModuleListrangeZnum_hidden_layerslayersrrYrZrMr!rQAs  zBlipEncoder.__init__N)rroutput_hidden_states return_dictrc s dk r n|jj|dk r |n|jj}|dk r4|n|jj}|rDdnd}rPdnd}|}t|jD]n\} } |rx||f}|jr|jrfdd} tj j | | ||} n| ||d} | d}rb|| df}qb|r||f}|st dd |||fDSt |||d S) a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Embedded representation of the inputs. Should be float, not int tokens. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr csfdd}|S)Ncs|fSrr )inputs)rrr r!custom_forwardsszJBlipEncoder.forward..create_custom_forward..custom_forwardr )rrr)rr!create_custom_forwardrsz2BlipEncoder.forward..create_custom_forwardrrrcss|]}|dk r|VqdSrr )rGvr r r!rIsz&BlipEncoder.forward..)r)r*r+) rNrruse_return_dict enumeraterrZtrainingrutils checkpointrJr ) r0rvrrrrZencoder_statesZall_attentionsr*idxZ encoder_layerrZ layer_outputsr rr!rlGsD      zBlipEncoder.forward)NNNN)r2r3r4r5rrQrrrmrrrr rlrnr r rZr!r7s   rc sxeZdZdZeZedfdd Zeee e edd e e j e ee ee eeee fddd Zd d ZZS) BlipVisionModelr\rMcsJt|||_|j}t||_t||_tj ||j d|_ | dSr) rPrQrNrRrLrkrencoderrrrpost_layernorm post_initrtrZr r!rQs   zBlipVisionModel.__init__ output_typerN)r\rrrrc Cs|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|dkrLtd||}|j||||d}|d}||}|dddddf}||}|s||f|ddSt|||j |j dS)z Returns: Nz You have to specify pixel_values)rvrrrrr)r)Z pooler_outputr*r+) rNrrrr|rkrrr r*r+) r0r\rrrr*Zencoder_outputsr) pooled_outputr r r!rls2    zBlipVisionModel.forwardcCs|jSr)rkr/r r r!get_input_embeddingssz$BlipVisionModel.get_input_embeddings)NNNN)r2r3r4main_input_namerrrQrBLIP_VISION_INPUTS_DOCSTRINGrr rrr6rrrrlrrnr r rZr!rs"   -rcseZdZeZedfdd Zeedee j ee j ee j ee e j dddZ eedee j ee e j dd d Zeeeeed dee jee j ee j ee jee ee ee ee eeefd d dZZS) BlipModelrMcst|t|jts.tdt|jdt|jtsPtdt|jd|j}|j}|j |_ |j |_ |j |_ t ||_t||_tj|j |j dd|_tj|j |j dd|_tt|jj|_|dS)NzKconfig.text_config is expected to be of type BlipTextConfig but is of type .zOconfig.vision_config is expected to be of type BlipVisionConfig but is of type F)r)rPrQr text_configrr|typerrZprojection_dimrRZtext_embed_dimZvision_embed_dimr text_modelr vision_modelrrvisual_projectiontext_projectionrTrrrNZlogit_scale_init_value logit_scaler)r0rNrrrZr r!rQs(     zBlipModel.__init__N)rurrprrcCs<|dk r |n|jj}|j||||d}|d}||}|S)a Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`BlipTextModel`]. Examples: ```python >>> from transformers import AutoProcessor, BlipModel >>> model = BlipModel.from_pretrained("Salesforce/blip-image-captioning-base") >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base") >>> inputs = processor(text=["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> text_features = model.get_text_features(**inputs) ```N)rurrprr)rNrrr)r0rurrpr text_outputsrZ text_featuresr r r!get_text_featuress zBlipModel.get_text_features)r\rrcCs8|dk r |n|jj}|j||d}|d}||}|S)aV Returns: image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`BlipVisionModel`]. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, BlipModel >>> model = BlipModel.from_pretrained("Salesforce/blip-image-captioning-base") >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> image_features = model.get_image_features(**inputs) ```N)r\rr)rNrrr)r0r\rvision_outputsrZimage_featuresr r r!get_image_featuress  zBlipModel.get_image_featuresr) rur\rrp return_lossrrrrc  Cs*|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|j||||d} |j||||||d} | d} || } | d} || } | | jdddd} | | jdddd} |j } t | | | }| }d}|rt|}|s||| | | | f}|dk r|f|S|St|||| | | | d S) a Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, BlipModel >>> model = BlipModel.from_pretrained("Salesforce/blip-image-captioning-base") >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor( ... text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True ... ) >>> outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities ```Nr\rrr)rurrprrrrrOr_T)praZkeepdim)r'r?r@rAr(rBrC)rNrrrrrrrZnormrexprrr$r%r>)r0rur\rrprrrrrrr(rArr@r?r'rr r r!rlDsT&    zBlipModel.forward)NNNN)NN)NNNNNNNN)r2r3r4rrrQrBLIP_TEXT_INPUTS_DOCSTRINGrrrmrr6rrrBLIP_INPUTS_DOCSTRINGrr>rxrrrlrnr r rZr!rsV %$  ra BLIP Model for image captioning. The model consists of a vision encoder and a text decoder. One can optionally pass `input_ids` to the model, which serve as a text prompt, to make the text decoder continue the prompt. Otherwise, the decoder starts generating text from the [BOS] (beginning-of-sequence) token. will start generating the caption from the text input. If no text input is provided, the decoder will start with the [BOS] token only. c seZdZeZdgZdZedfdd Zej dddZ e e e eed dejeejeejeeeeeejeeeeefd d d ZedejeejeejejdddZZS)BlipForConditionalGeneration)text_decoder.cls.predictions.decoder.biasr\rMcsDt|t|j|_t|j|_|jj|_ |jj |_ | dSr) rPrQrrrrr text_decoder bos_token_iddecoder_input_ids pad_token_iddecoder_pad_token_idrrYrZr r!rQs      z%BlipForConditionalGeneration.__init__rDcCs |jjjSrrrkrWr/r r r!rsz1BlipForConditionalGeneration.get_input_embeddingsrN)r\rurrrlabelsrrc Cs|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|j||||d}|d} |j||| ||dd} |s| d| d| |df|dd} tdd | DSt| j| j | |j |j |j d S) a Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, BlipForConditionalGeneration >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base") >>> model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "A picture of" >>> inputs = processor(images=image, text=text, return_tensors="pt") >>> outputs = model(**inputs) ```Nrrr)rurencoder_hidden_statesrr reductionrrOcss|]}|dk r|VqdSrr rGrr r r!rIsz7BlipForConditionalGeneration.forward..)r'rr(r)r*r+) rNrrrrrrJr&r'rr)r*r+) r0r\rurrrrrrr(rr r r!rls<" $z$BlipForConditionalGeneration.forward)r\rurrc Ks|jd}|j|d}|d}tj|ddtjd|j}t|t rXt |}n0|dkrt |j |j j jgg|d|j}|j j j|dddf<|dk r|ddddfnd}|jjf|ddddf|j j j|j j j|||d|} | S)a Overrides *generate* function to be able to use the model as a conditional generator Parameters: pixel_values (*torch.FloatTensor* of shape *(batch_size, num_channels, image_height, image_width)*: Input image to be processed input_ids (*torch.LongTensor* of shape *(batch_size, sequence_length)*, *optional*): The sequence used as a prompt for the generation. attention_mask (*torch.LongTensor* of shape *(batch_size, sequence_length)*, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, BlipForConditionalGeneration >>> model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base") >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model.generate(**inputs) >>> print(processor.decode(outputs[0], skip_special_tokens=True)) two cats sleeping on a couch ``` rr\Nr_r]r)ru eos_token_idrrrencoder_attention_mask)rbrronesrilongrdrrlistrxrrNrrrepeatrrgenerate sep_token_idr) r0r\rurgenerate_kwargsrjrr(image_attention_maskrr r r!rs6(  $    z%BlipForConditionalGeneration.generate)NNNNNN)NN)r2r3r4rr_tied_weights_keysrrQrModulerrrrr&rrr6rrxrrrrlno_gradrrnr r rZr!rs@    EraS BLIP Model for visual question answering. The model consists of a vision encoder, a text encoder as well as a text decoder. The vision encoder will encode the input image, the text encoder will encode the input question together with the encoding of the image, and the text decoder will output the answer to the question. cseZdZeZdgZedfdd ZejdddZ e e e e eddejejeejeejeejeeeeeejeeeee fd d d Zedejejeejejd ddZZS)BlipForQuestionAnsweringrrMcsTt|t|j|_t|jdd|_t|j|_ |jj |_ |jj |_ |dS)NFZadd_pooling_layer)rPrQrrrrr text_encoderrrrrrdecoder_start_token_idrrYrZr r!rQ[s     z!BlipForQuestionAnswering.__init__rDcCs |jjjSrrr/r r r!rjsz-BlipForQuestionAnswering.get_input_embeddingsrN) rur\rdecoder_attention_maskrrrrrrc  CsP|dkr|dkrtd| dk r$| n|jj} |dk r8|n|jj}|dk rL|n|jj}|j|||| d} | d} tj| ddtj d} |j ||| | | d} |dk r|dkr|}| s| dn| j } |j ||| ||| dd }|dk r| r|j n |d}nd}| s8|| | df| d d}td d |DSt|| | j | j| jd S)a Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, BlipForQuestionAnswering >>> model = BlipForQuestionAnswering.from_pretrained("Salesforce/blip-vqa-base") >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-vqa-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> # training >>> text = "How many cats are in the picture?" >>> label = "2" >>> inputs = processor(images=image, text=text, return_tensors="pt") >>> labels = processor(text=label, return_tensors="pt").input_ids >>> inputs["labels"] = labels >>> outputs = model(**inputs) >>> loss = outputs.loss >>> loss.backward() >>> # inference >>> text = "How many cats are in the picture?" >>> inputs = processor(images=image, text=text, return_tensors="pt") >>> outputs = model.generate(**inputs) >>> print(processor.decode(outputs[0], skip_special_tokens=True)) 2 ```NaEither `decoder_input_ids` or `labels` should be passed when calling `forward` with `BlipForQuestionAnswering`. if you are training the model make sure that `labels` is passed, if you are using the model for inference make sure that `decoder_input_ids` is passed or call `generate`rrr_r]rurrrrr)rurrrrrrrOcss|]}|dk r|VqdSrr rr r r!rIsz3BlipForQuestionAnswering.forward..)r'r(r)r*r+)r|rNrrrrrrrirrr)rr'rrJr9r*r+)r0rur\rrrrrrrrr(rr=Z answer_outputZ decoder_lossrr r r!rlms`0 z BlipForQuestionAnswering.forward)rur\rrc Ks|j|d}|d}tj|ddtjd|j}t|trLt |}|j ||||dd}|d} tj| ddtjd| j} tj | ddf|j | jd } |j jf| |jjj|jjj| | d |} | S) aD Overrides *generate* function to be able to use the model as a conditional generator Parameters: input_ids (*torch.LongTensor* of shape *(batch_size, sequence_length)*): The sequence used as a prompt for the generation. pixel_values (*torch.FloatTensor* of shape *(batch_size, num_channels, image_height, image_width)*: Input image to be processed attention_mask (*torch.LongTensor* of shape *(batch_size, sequence_length)*, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`. `1` for tokens that are NOT MASKED, `0` for MASKED tokens. **generate_kwargs: Additional arguments passed to the *generate* function of the decoder Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, BlipForQuestionAnswering >>> model = BlipForQuestionAnswering.from_pretrained("Salesforce/blip-vqa-base") >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-vqa-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "How many cats are in the picture?" >>> inputs = processor(images=image, text=text, return_tensors="pt") >>> outputs = model.generate(**inputs) >>> print(processor.decode(outputs[0], skip_special_tokens=True)) 2 ``` rrNr_r]Frr)Z fill_valuer)rurrrr)rrrrirrdrrrrxrfullrrrrNrrr) r0rur\rrrr(rZquestion_outputsr=Zquestion_attention_maskZbos_idsrr r r!rs:+ $  $  z!BlipForQuestionAnswering.generate)NNNNNNN)N)r2r3r4rrrrQrrrrrrr9rrrxr6rrrrrlrrrnr r rZr!rOsB   nra BLIP Model with a vision and text projector, and a classification head on top. The model is used in the context of image-text retrieval. Given an image and a text, the model returns the probability of the text being relevant to the image. c seZdZeZedfdd ZejdddZe e e e e dd ejejeeeejeeeeeeeee fd d d ZZS)BlipForImageTextRetrievalrMcst|t|j|_t|jdd|_t |jj |j |_ t |jj |j |_ t |jj d|_t|dst|jjn|j|_t|ds|jjn|j|_|dS)NFrrOrr)rPrQrrrrrrrrrRZimage_text_hidden_size vision_proj text_projitm_headrrrrrrrYrZr r!rQ8s    z"BlipForImageTextRetrieval.__init__rDcCs |jjjSrrr/r r r!rVsz.BlipForImageTextRetrieval.get_input_embeddingsrTN)rur\ use_itm_headrrrrrcCs~|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|j||||d}|d} tj| ddtjd} |r|j ||| | |d} |s| dn| j } | | dddddf} nv|j |||d} |s| dn| j } t | | dddddfdd} t || dddddfdd}| |} |sf| |df|d d| f}td d |DSt| |j |j|j| d S) a Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, BlipForImageTextRetrieval >>> model = BlipForImageTextRetrieval.from_pretrained("Salesforce/blip-itm-base-coco") >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-itm-base-coco") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "an image of a cat" >>> inputs = processor(images=image, text=text, return_tensors="pt") >>> outputs = model(**inputs) ``` Nrrr_r]r)rurrr`rOcss|]}|dk r|VqdSrr rr r r!rIsz4BlipForImageTextRetrieval.forward..)r;r)r*r+r=)rNrrrrrrrirrr)rrrrr$rJr:r*r+)r0rur\rrrrrrr(Z image_attsr=rZ image_featZ text_featrr r r!rlYsR!$$ z!BlipForImageTextRetrieval.forward)TNNNN)r2r3r4rrrQrrrrrrr9rrrxr6rrrrrlrnr r rZr!r-s(   r)?r5r, dataclassesrtypingrrrrrZtorch.utils.checkpointrZtorch.nn.functionalrZ activationsr Zmodeling_outputsr r Zmodeling_utilsr rrrrrrZconfiguration_bliprrrZmodeling_blip_textrrZ get_loggerr2loggerZ_CHECKPOINT_FOR_DOCZ"BLIP_PRETRAINED_MODEL_ARCHIVE_LISTrmr"r%r&r9r:r>rrLroryrrrZBLIP_START_DOCSTRINGrrrrrrrrrr r r r!s      , *$ !F0,%\BM %W