U ,-ev@sdZddlZddlmZmZmZmZddlZddlZddlm Z ddl m Z m Z ddl mZddlmZmZmZdd lmZdd lmZmZdd lmZmZmZmZd d lmZee Z!dZ"dZ#dgZ$Gddde j%Z&Gddde j%Z'Gddde j%Z(Gddde j%Z)Gddde j%Z*Gddde j%Z+Gddde j%Z,Gddde j%Z-Gd d!d!e j%Z.Gd"d#d#e j%Z/Gd$d%d%eZ0d&Z1d'Z2ed(e1Gd)d*d*e0Z3ed+e1Gd,d-d-e0Z4dS).z PyTorch ViViT model.N)OptionalSetTupleUnion)nn)CrossEntropyLossMSELoss)ACT2FN)BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutput)PreTrainedModel) find_pruneable_heads_and_indicesprune_linear_layer)add_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) VivitConfigzgoogle/vivit-b-16x2-kinetics400rcs(eZdZdZfddZddZZS)VivitTubeletEmbeddingsa Construct Vivit Tubelet embeddings. This module turns a batch of videos of shape (batch_size, num_frames, num_channels, height, width) into a tensor of shape (batch_size, seq_len, hidden_size) to be consumed by a Transformer encoder. The seq_len (the number of patches) equals (number of frames // tubelet_size[0]) * (height // tubelet_size[1]) * (width // tubelet_size[2]). cs|t|j|_|j|_|j|_|j|jd|j|jd|j|jd|_|j|_t j |j |j|j|jd|_ dS)Nrr)Z kernel_sizeZstride) super__init__ num_frames image_sizeZ tubelet_sizeZ patch_size num_patches hidden_sizeZ embed_dimrConv3d num_channels projectionselfconfig __class__i/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/models/vivit/modeling_vivit.pyr8s" zVivitTubeletEmbeddings.__init__c Cs|j\}}}}}||jks$||jkrJtd|d|d|jd|jd |ddddd }||}||ddd}|S) NzInput image size (*z) doesn't match model (z).rrrr )shaper ValueErrorpermuter!flatten transpose)r# pixel_values batch_sizerr heightwidthxr'r'r(forwardHs  zVivitTubeletEmbeddings.forward__name__ __module__ __qualname____doc__rr5 __classcell__r'r'r%r(r-s rcs(eZdZdZfddZddZZS)VivitEmbeddingsz Vivit Embeddings. Creates embeddings from a video using VivitTubeletEmbeddings, adds CLS token and positional embeddings. csdtttdd|j|_t||_ ttd|j j d|j|_ t |j |_||_dS)Nr)rrr ParametertorchZzerosr cls_tokenrpatch_embeddingsrposition_embeddingsDropouthidden_dropout_probdropoutr$r"r%r'r(r_s  zVivitEmbeddings.__init__cCsP|jd}||}|j|ddg}tj||fdd}||j}||}|SNrrdim)r+r@r?Ztiler>catrArD)r#r0r1 embeddingsZ cls_tokensr'r'r(r5ks    zVivitEmbeddings.forwardr6r'r'r%r(r<Xs r<csleZdZeddfdd ZejejdddZd eeje e e ejejfe ejfd d d Z Z S) VivitSelfAttentionNr$returncst|j|jdkr@t|ds@td|jfd|jd|j|_t|j|j|_|j|j|_t j |j|j|j d|_ t j |j|j|j d|_ t j |j|j|j d|_t |j|_dS)NrZembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .)bias)rrrnum_attention_headshasattrr,intattention_head_size all_head_sizerLinearZqkv_biasquerykeyvaluerBZattention_probs_dropout_probrDr"r%r'r(r}s zVivitSelfAttention.__init__)r4rLcCs6|dd|j|jf}||}|ddddS)Nrrrr )sizerOrRviewr-)r#r4Z new_x_shaper'r'r(transpose_for_scoress z'VivitSelfAttention.transpose_for_scoresF) head_maskoutput_attentionsrLc Cs||}|||}|||}||}t||dd}|t|j }t j j |dd} | | } |dk r| |} t| |} | dddd} | dd|jf} | | } |r| | fn| f} | S)NrXrFrrrr )rUr[rVrWr>matmulr/mathsqrtrRrZ functionalZsoftmaxrDr- contiguousrYrSrZ) r# hidden_statesr\r]Zmixed_query_layerZ key_layerZ value_layerZ query_layerZattention_scoresZattention_probsZ context_layerZnew_context_layer_shapeoutputsr'r'r(r5s      zVivitSelfAttention.forward)NF)r7r8r9rrr>Tensorr[rboolrrr5r;r'r'r%r(rJ|srJcs@eZdZdZeddfdd ZejejejdddZZ S) VivitSelfOutputz The residual connection is defined in VivitLayer instead of here (as is the case with other models), due to the layernorm applied before each block. NrKcs.tt|j|j|_t|j|_dSN) rrrrTrdenserBrCrDr"r%r'r(rs zVivitSelfOutput.__init__)rc input_tensorrLcCs||}||}|SrhrirDr#rcrjr'r'r(r5s  zVivitSelfOutput.forward) r7r8r9r:rrr>rer5r;r'r'r%r(rgsrgcspeZdZeddfdd ZeeddddZd ej e ej e e e ej ej fe ej fd d d ZZS) VivitAttentionNrKcs*tt||_t||_t|_dSrh)rrrJ attentionrgoutputset pruned_headsr"r%r'r(rs   zVivitAttention.__init__)headsrLcCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dSrE)lenrrnrOrRrqrrUrVrWrorirSunion)r#rrindexr'r'r( prune_headss zVivitAttention.prune_headsF)rcr\r]rLcCs4||||}||d|}|f|dd}|S)Nrr)rnro)r#rcr\r]Z self_outputsattention_outputrdr'r'r(r5szVivitAttention.forward)NF)r7r8r9rrrrQrvr>rerrfrrr5r;r'r'r%r(rmsrmcs$eZdZfddZddZZS)VivitIntermediatecsPtt|j|j|_t|j|_ t |j t rDt |j |_n|j |_dSrh)rrrrTrintermediate_sizerirBrCrD isinstanceZ hidden_actstrr intermediate_act_fnr"r%r'r(rs   zVivitIntermediate.__init__cCs"||}||}||}|Srh)rir|rD)r#rcr'r'r(r5s   zVivitIntermediate.forwardr7r8r9rr5r;r'r'r%r(rxs rxcs$eZdZfddZddZZS) VivitOutputcs.tt|j|j|_t|j|_ dSrh) rrrrTryrrirBrCrDr"r%r'r(rs zVivitOutput.__init__cCs ||}||}||}|Srhrkrlr'r'r(r5 s  zVivitOutput.forwardr}r'r'r%r(r~s r~cs*eZdZdZfddZdddZZS) VivitLayerzNThis corresponds to the EncoderBlock class in the scenic/vivit implementation.csbt|j|_d|_t||_t||_t||_ t j |j |j d|_t j |j |j d|_dS)NrZeps)rrZchunk_size_feed_forwardZ seq_len_dimrmrnrx intermediater~ror LayerNormrlayer_norm_epslayernorm_beforelayernorm_afterr"r%r'r(rs    zVivitLayer.__init__NFcCs`|j||||d}|d}|dd}||}||}||}|||}|f|}|S)Nr]rr)rnrrrro)r#rcr\r]Zself_attention_outputsrwrdZ layer_outputr'r'r(r5"s     zVivitLayer.forward)NFr6r'r'r%r(rs rcs&eZdZfddZdddZZS) VivitEncodercs:t|_tfddtjD|_d|_dS)Ncsg|] }tqSr')r).0_r$r'r( @sz)VivitEncoder.__init__..F) rrr$rZ ModuleListrangenum_hidden_layerslayergradient_checkpointingr"r%rr(r=s  zVivitEncoder.__init__NFTc s|rdnd}rdnd}t|jD]\}} |r8||f}|dk rH||nd} |jr||jr|fdd} tjj| | || } n | || } | d}r"|| df}q"|r||f}|stdd|||fDSt|||dS) Nr'csfdd}|S)Ncs|fSrhr')inputs)moduler]r'r(custom_forwardWszKVivitEncoder.forward..create_custom_forward..custom_forwardr')rrr)rr(create_custom_forwardVsz3VivitEncoder.forward..create_custom_forwardrrcss|]}|dk r|VqdSrhr')rvr'r'r( msz'VivitEncoder.forward..)last_hidden_staterc attentions) enumeraterrZtrainingr>utils checkpointtupler ) r#rcr\r]output_hidden_states return_dictZall_hidden_statesZall_self_attentionsiZ layer_moduleZlayer_head_maskrZ layer_outputsr'rr(r5Cs4       zVivitEncoder.forward)NFFTr}r'r'r%r(r<s  rcs$eZdZfddZddZZS) VivitPoolercs*tt|j|j|_t|_dSrh)rrrrTrriZTanh activationr"r%r'r(rvs zVivitPooler.__init__cCs(|dddf}||}||}|S)Nr)rir)r#rcZfirst_token_tensor pooled_outputr'r'r(r5{s  zVivitPooler.forwardr}r'r'r%r(rus rc@s2eZdZdZeZdZdZdZddZ d dd Z d S) VivitPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. vivitr0TcCst|tjtjfr@|jjjd|jjd|j dk r|j j nt|tj r|jjjd|jjd|j dk r|jj|j  nHt|tj r|j j |jjdn t|tjr|jjd|jjddS)zInitialize the weightsg)ZmeanZstdNg?)rzrrTrweightdataZnormal_r$Zinitializer_rangerNZzero_Z EmbeddingZ padding_idxrZfill_r=)r#rr'r'r( _init_weightss      z"VivitPreTrainedModel._init_weightsFcCst|tr||_dSrh)rzrr)r#rrWr'r'r(_set_gradient_checkpointings z0VivitPreTrainedModel._set_gradient_checkpointingN)F) r7r8r9r:r config_classZbase_model_prefixZmain_input_nameZsupports_gradient_checkpointingrrr'r'r'r(rsraG This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`VivitConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a\ Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_frames, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`VivitImageProcessor`]. See [`VivitImageProcessor.preprocess`] for details. head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z_The bare ViViT Transformer model outputting raw hidden-states without any specific head on top.c seZdZd fdd ZddZddZeeee e dde e j e e j e ee ee eeee j e fd d d ZZS) VivitModelTcsXt|||_t||_t||_tj|j |j d|_ |rFt |nd|_ |dS)Nr)rrr$r<rIrencoderrrrr layernormrpooler post_init)r#r$add_pooling_layerr%r'r(rs   zVivitModel.__init__cCs|jjSrh)rIr@)r#r'r'r(get_input_embeddingsszVivitModel.get_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. Args: heads_to_prune: dict of {layer_num: list of heads to prune in this layer} N)itemsrrrnrv)r#Zheads_to_prunerrrr'r'r( _prune_headsszVivitModel._prune_heads output_typerN)r0r\r]rrrLc Cs|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|dkrLtd|||jj}||}|j|||||d}|d}| |}|j dk r| |nd} |s|| f|ddSt || |j |j dS)a Returns: Examples: ```python >>> import av >>> import numpy as np >>> from transformers import VivitImageProcessor, VivitModel >>> from huggingface_hub import hf_hub_download >>> np.random.seed(0) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`List[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`List[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # video clip consists of 300 frames (10 seconds at 30 FPS) >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample 32 frames >>> indices = sample_frame_indices(clip_len=32, frame_sample_rate=1, seg_len=container.streams.video[0].frames) >>> video = read_video_pyav(container=container, indices=indices) >>> image_processor = VivitImageProcessor.from_pretrained("google/vivit-b-16x2-kinetics400") >>> model = VivitModel.from_pretrained("google/vivit-b-16x2-kinetics400") >>> # prepare video for the model >>> inputs = image_processor(list(video), return_tensors="pt") >>> # forward pass >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 3137, 768] ```Nz You have to specify pixel_valuesr\r]rrrr)rZ pooler_outputrcr)r$r]ruse_return_dictr,Z get_head_maskrrIrrrr rcr) r#r0r\r]rrZembedding_outputZencoder_outputssequence_outputrr'r'r(r5s4V  zVivitModel.forward)T)NNNNN)r7r8r9rrrrVIVIT_INPUTS_DOCSTRINGrr _CONFIG_FOR_DOCrr> FloatTensorrfrrr5r;r'r'r%r(rs$   rzViViT Transformer model with a video classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for Kinetics-400.c sxeZdZfddZeeeeedde e j e e j e e j e e e e e e eee j efdddZZS) VivitForVideoClassificationcsRt||j|_t|dd|_|jdkr 1` a classification loss is computed (Cross-Entropy). Returns: Examples: ```python >>> import av >>> import numpy as np >>> import torch >>> from transformers import VivitImageProcessor, VivitForVideoClassification >>> from huggingface_hub import hf_hub_download >>> np.random.seed(0) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`List[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`List[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # video clip consists of 300 frames (10 seconds at 30 FPS) >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample 32 frames >>> indices = sample_frame_indices(clip_len=32, frame_sample_rate=4, seg_len=container.streams.video[0].frames) >>> video = read_video_pyav(container=container, indices=indices) >>> image_processor = VivitImageProcessor.from_pretrained("google/vivit-b-16x2-kinetics400") >>> model = VivitForVideoClassification.from_pretrained("google/vivit-b-16x2-kinetics400") >>> inputs = image_processor(list(video), return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) ... logits = outputs.logits >>> # model predicts one of the 400 Kinetics-400 classes >>> predicted_label = logits.argmax(-1).item() >>> print(model.config.id2label[predicted_label]) LABEL_116 ```NrrrrXr)losslogitsrcr) r$rrrrrrZrr rcr) r#r0r\rr]rrrdrrrZloss_fctror'r'r(r5ss4_ z#VivitForVideoClassification.forward)NNNNNN)r7r8r9rrrrr rrr>rZ LongTensorrfrrr5r;r'r'r%r(ras$  r)5r:r`typingrrrrr>Ztorch.utils.checkpointrZtorch.nnrrZ activationsr Zmodeling_outputsr r r Zmodeling_utilsrZ pytorch_utilsrrrrrrrZconfiguration_vivitrZ get_loggerr7loggerZ_CHECKPOINT_FOR_DOCrZ#VIVIT_PRETRAINED_MODEL_ARCHIVE_LISTModulerr<rJrgrmrxr~rrrrZVIVIT_START_DOCSTRINGrrrr'r'r'r(sR     +$=''9"