U 9%e@sRdZddlZddlZddlmZddlmZddlm Z m Z m Z m Z ddl ZddlZddlZddlmZddlmZmZmZdd lmZdd lmZmZdd lmZdd lmZmZdd l m!Z!m"Z"m#Z#m$Z$m%Z%ddl&m'Z'm(Z(ddl)m*Z*e$+e,Z-dZ.dZ/dgZ0eGddde!Z1eGddde!Z2ddZ3Gdddej4Z5Gdddej4Z6Gdddej4Z7Gdd d ej4Z8Gd!d"d"ej4Z9Gd#d$d$ej4Z:Gd%d&d&ej4Z;Gd'd(d(ej4Zd-Z?d.Z@e"d/e?Gd0d1d1e>ZAGd2d3d3ej4ZBe"d4e?Gd5d6d6e>ZCe"d7e?Gd8d9d9e>ZDdS):z- PyTorch VideoMAE (masked autoencoder) model.N)deepcopy) dataclass)OptionalSetTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BaseModelOutputImageClassifierOutput)PreTrainedModel) find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings)IMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD)VideoMAEConfigrzMCG-NJU/videomae-basec@sLeZdZUdZdZejed<dZe e ejed<dZ e e ejed<dS)VideoMAEDecoderOutputaO Class for VideoMAEDecoder's outputs, with potential hidden states and attentions. Args: logits (`torch.FloatTensor` of shape `(batch_size, patch_size ** 2 * num_channels)`): Pixel reconstruction logits. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlogits hidden_states attentions) __name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rrrrr'r'm/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/transformers/models/videomae/modeling_videomae.pyr8s rc@s^eZdZUdZdZeejed<dZ ejed<dZ ee ejed<dZ ee ejed<dS)VideoMAEForPreTrainingOutputa Class for VideoMAEForPreTraining's outputs, with potential hidden states and attentions. Args: loss (`torch.FloatTensor` of shape `(1,)`): Pixel reconstruction loss. logits (`torch.FloatTensor` of shape `(batch_size, patch_size ** 2 * num_channels)`): Pixel reconstruction logits. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlossrrr) r r!r"r#r*rr$r%r&rrrrr'r'r'r(r)Os r)csfddtfddt|D}t|dddddf|dddddf<t|dddddf|dddddf<t|dS) z Sinusoid position encoding tablecsfddtDS)Nc s(g|] }tdd|dqS)i')nppower).0Zhid_j)d_hidpositionr'r( pszOget_sinusoid_encoding_table..get_position_angle_vec..)ranger0)r/r3r(get_position_angle_vecosz;get_sinusoid_encoding_table..get_position_angle_veccsg|] }|qSr'r')r.Zpos_i)r4r'r(r1rsz/get_sinusoid_encoding_table..Nrr+r)r,arrayr2sincosr$r%Z unsqueeze)Z n_positionr/Zsinusoid_tabler')r/r4r(get_sinusoid_encoding_tableks  ..r8cs(eZdZdZfddZddZZS)VideoMAEEmbeddingsz7 Construct the patch and position embeddings. cs8tt||_|jj|_t|j|j|_||_dSN) super__init__VideoMAEPatchEmbeddingspatch_embeddings num_patchesr8 hidden_sizeposition_embeddingsconfigselfrB __class__r'r(r<s    zVideoMAEEmbeddings.__init__cCsZ||}||j||j}|dk rV|j\}}}||}||d|}|S)N) r>rAtype_astodeviceclonedetachshapereshape)rD pixel_valuesbool_masked_pos embeddings batch_size_ num_channelsr'r'r(forwards    zVideoMAEEmbeddings.forwardr r!r"r#r<rU __classcell__r'r'rEr(r9ys r9cs(eZdZdZfddZddZZS)r=aw Video to Patch Embedding. This module turns a batch of videos of shape (batch_size, num_frames, num_channels, height, width) into a tensor of shape (batch_size, seq_len, hidden_size) to be consumed by a Transformer encoder. The seq_len (the number of patches) equals (number of frames // tubelet_size) * (height // patch_size) * (width // patch_size). c st|j}|j}|j}|j}|j}|j}t|t j j r@|n||f}t|t j j rZ|n||f}||_||_t ||_|d|d|d|d||j}||_||_ tj|||j|d|df|j|d|dfd|_dS)Nrr)Z in_channelsZ out_channelsZ kernel_sizeZstride)r;r< image_size patch_sizerTr@ num_frames tubelet_size isinstance collectionsabcIterableintr?rConv3d projection) rDrBrXrYrTr@rZr[r?rEr'r(r<s,  (z VideoMAEPatchEmbeddings.__init__c Cs|j\}}}}}||jkr"td||jdks>||jdkrltd|d|d|jdd|jdd |dddd d }||ddd}|S) NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.rrzInput image size (*z) doesn't match model (z).r+r )rMrT ValueErrorrXpermuterbflatten transpose)rDrOrRrZrTheightwidthrQr'r'r(rUs (zVideoMAEPatchEmbeddings.forwardrVr'r'rEr(r=s r=csleZdZeddfdd ZejejdddZd eeje e e ejejfe ejfd d d Z Z S) VideoMAESelfAttentionNrBreturncst|j|jdkr@t|ds@td|jfd|jd|j|_t|j|j|_|j|j|_t j |j|jdd|_ t j |j|jdd|_ t j |j|jdd|_ |jrt t|j|_t t|j|_n d|_d|_t |j|_dS)NrZembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .Fbias)r;r<r@num_attention_headshasattrrer`attention_head_size all_head_sizerLinearquerykeyvalueZqkv_bias Parameterr$zerosq_biasv_biasDropoutZattention_probs_dropout_probdropoutrCrEr'r(r<s" zVideoMAESelfAttention.__init__)xrmcCs6|dd|j|jf}||}|ddddS)NrGrr+rr )sizerqrsviewrf)rDrZ new_x_shaper'r'r(transpose_for_scoress z*VideoMAESelfAttention.transpose_for_scoresF) head_maskoutput_attentionsrmcCs*|jdk rtj|jddnd}tjj||jj|d}tjj||j j|jd}tjj||j j|jd}| |}| |} | |} t | | dd} | t|j} tjj| dd} || } |dk r| |} t | | } | ddd d } | dd|jf}| |} |r | | fn| f}|S) NF)Z requires_grad)inputweightrprGdimrr+rr )r{r$Z zeros_liker|rZ functionalZlinearrwrrxrvrmatmulrhmathsqrtrsZsoftmaxr~rf contiguousrrtr)rDrrrZk_biaskeysvaluesZqueriesZ key_layerZ value_layerZ query_layerZattention_scoresZattention_probsZ context_layerZnew_context_layer_shapeoutputsr'r'r(rUs&      zVideoMAESelfAttention.forward)NF)r r!r"rr<r$TensorrrboolrrrUrWr'r'rEr(rksrkcs@eZdZdZeddfdd ZejejejdddZZ S) VideoMAESelfOutputz The residual connection is defined in VideoMAELayer instead of here (as is the case with other models), due to the layernorm applied before each block. Nrlcs.tt|j|j|_t|j|_dSr:) r;r<rrur@denser}hidden_dropout_probr~rCrEr'r(r<s zVideoMAESelfOutput.__init__r input_tensorrmcCs||}||}|Sr:rr~rDrrr'r'r(rU s  zVideoMAESelfOutput.forward) r r!r"r#rr<r$rrUrWr'r'rEr(rsrcspeZdZeddfdd ZeeddddZd ej e ej e e e ej ej fe ej fd d d ZZS) VideoMAEAttentionNrlcs*tt||_t||_t|_dSr:)r;r<rk attentionroutputset pruned_headsrCrEr'r(r<)s   zVideoMAEAttention.__init__)headsrmcCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr)lenrrrqrsrrrvrwrxrrrtunion)rDrindexr'r'r( prune_heads/s zVideoMAEAttention.prune_headsFrrrrmcCs4||||}||d|}|f|dd}|S)Nrr)rr)rDrrrZ self_outputsattention_outputrr'r'r(rUAszVideoMAEAttention.forward)NF)r r!r"rr<rr`rr$rrrrrrUrWr'r'rEr(r(srcs8eZdZeddfdd ZejejdddZZS)VideoMAEIntermediateNrlcsBtt|j|j|_t|jt r6t |j|_ n|j|_ dSr:) r;r<rrur@intermediate_sizerr\Z hidden_actstrr intermediate_act_fnrCrEr'r(r<Qs   zVideoMAEIntermediate.__init__)rrmcCs||}||}|Sr:)rr)rDrr'r'r(rUYs  zVideoMAEIntermediate.forward r r!r"rr<r$rrUrWr'r'rEr(rPsrcs<eZdZeddfdd ZejejejdddZZS)VideoMAEOutputNrlcs.tt|j|j|_t|j|_ dSr:) r;r<rrurr@rr}rr~rCrEr'r(r<bs zVideoMAEOutput.__init__rcCs ||}||}||}|Sr:rrr'r'r(rUgs  zVideoMAEOutput.forwardrr'r'rEr(rasrcs`eZdZdZeddfdd Zd ejeeje e e ejejfe ejfddd Z Z S) VideoMAELayerz?This corresponds to the Block class in the timm implementation.Nrlcsbt|j|_d|_t||_t||_t||_ t j |j |j d|_t j |j |j d|_dS)NrZeps)r;r<Zchunk_size_feed_forwardZ seq_len_dimrrr intermediaterrr LayerNormr@layer_norm_epslayernorm_beforelayernorm_afterrCrEr'r(r<ts    zVideoMAELayer.__init__FrcCs`|j||||d}|d}|dd}||}||}||}|||}|f|}|S)Nrrr)rrrrr)rDrrrZself_attention_outputsrrZ layer_outputr'r'r(rU~s     zVideoMAELayer.forward)NF)r r!r"r#rr<r$rrrrrrUrWr'r'rEr(rqs rc sNeZdZeddfdd Zd ejeejeeee e e fddd Z Z S) VideoMAEEncoderNrlcs:t|_tfddtjD|_d|_dS)Ncsg|] }tqSr'rr.rSrBr'r(r1sz,VideoMAEEncoder.__init__..F) r;r<rBr ModuleListr2num_hidden_layerslayergradient_checkpointingrCrErr(r<s  zVideoMAEEncoder.__init__FT)rrroutput_hidden_states return_dictrmc s|rdnd}rdnd}t|jD]\}} |r8||f}|dk rH||nd} |jr||jr|fdd} tjj| | || } n | || } | d}r"|| df}q"|r||f}|stdd|||fDSt|||dS) Nr'csfdd}|S)Ncs|fSr:r'inputsmodulerr'r(custom_forwardszNVideoMAEEncoder.forward..create_custom_forward..custom_forwardr'rrrrr(create_custom_forwardsz6VideoMAEEncoder.forward..create_custom_forwardrrcss|]}|dk r|VqdSr:r'r.vr'r'r( sz*VideoMAEEncoder.forward..Zlast_hidden_staterr) enumeraterrtrainingr$utils checkpointtupler) rDrrrrrall_hidden_statesall_self_attentionsi layer_moduleZlayer_head_maskr layer_outputsr'rr(rUs4       zVideoMAEEncoder.forward)NFFT)r r!r"rr<r$rrrrrrrUrWr'r'rEr(rs  rc@s2eZdZdZeZdZdZdZddZ d dd Z d S) VideoMAEPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. videomaerOTcCsjt|tjtjfr@|jjjd|jjd|j dk rf|j j n&t|tj rf|j j |jj ddS)zInitialize the weightsg)meanstdNg?) r\rrurardataZnormal_rBZinitializer_rangerpZzero_rZfill_)rDrr'r'r( _init_weightss   z%VideoMAEPreTrainedModel._init_weightsFcCst|tr||_dSr:)r\rr)rDrrxr'r'r(_set_gradient_checkpointings z3VideoMAEPreTrainedModel._set_gradient_checkpointingN)F) r r!r"r#r config_classZbase_model_prefixZmain_input_nameZsupports_gradient_checkpointingrrr'r'r'r(rs raJ This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`VideoMAEConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a\ Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_frames, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`VideoMAEImageProcessor.__call__`] for details. head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zbThe bare VideoMAE Model transformer outputting raw hidden-states without any specific head on top.c s~eZdZfddZddZddZeeee e dd e j e e je e je ee ee eeee fd d d ZZS) VideoMAEModelcsTt|||_t||_t||_|jr4d|_nt j |j |j d|_| dS)Nr)r;r<rBr9rQrencoderuse_mean_pooling layernormrrr@r post_initrCrEr'r(r<s   zVideoMAEModel.__init__cCs|jjSr:)rQr>)rDr'r'r(get_input_embeddings'sz"VideoMAEModel.get_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)rDZheads_to_prunerrr'r'r( _prune_heads*szVideoMAEModel._prune_heads output_typerNrOrPrrrrrmc Cs|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|||jj}|||}|j|||||d}|d} |jdk r|| } |s| f|ddSt | |j |j dS)a bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, sequence_length)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Each video in the batch must have the same number of masked patches. If `None`, then all patches are considered. Sequence length is `(num_frames // tubelet_size) * (image_size // patch_size) ** 2`. Returns: Examples: ```python >>> import av >>> import numpy as np >>> from transformers import AutoImageProcessor, VideoMAEModel >>> from huggingface_hub import hf_hub_download >>> np.random.seed(0) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`List[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`List[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # video clip consists of 300 frames (10 seconds at 30 FPS) >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample 16 frames >>> indices = sample_frame_indices(clip_len=16, frame_sample_rate=1, seg_len=container.streams.video[0].frames) >>> video = read_video_pyav(container, indices) >>> image_processor = AutoImageProcessor.from_pretrained("MCG-NJU/videomae-base") >>> model = VideoMAEModel.from_pretrained("MCG-NJU/videomae-base") >>> # prepare video for the model >>> inputs = image_processor(list(video), return_tensors="pt") >>> # forward pass >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 1568, 768] ```Nrrrrrrr) rBrruse_return_dictZ get_head_maskrrQrrrrr) rDrOrPrrrrZembedding_outputZencoder_outputssequence_outputr'r'r(rU2s.\   zVideoMAEModel.forward)NNNNN)r r!r"r<rrrVIDEOMAE_INPUTS_DOCSTRINGrr_CONFIG_FOR_DOCr$r%r BoolTensorrrrrrUrWr'r'rEr(rs&   rcs&eZdZfddZdddZZS)VideoMAEDecodercst|j|j|jd}t||j_|j_ |j _ |j _ tfddt|jD|_t|j|_|dkrt|j|nt|_d|_||_dS)Nr+csg|] }tqSr'rrZdecoder_configr'r(r1sz,VideoMAEDecoder.__init__..rF)r;r<rTr[rYrdecoder_hidden_sizer@Zdecoder_num_hidden_layersrZdecoder_num_attention_headsrqZdecoder_intermediate_sizerrrr2decoder_layersrnormruIdentityheadrrB)rDrBr?Zdecoder_num_labelsrErr(r<s zVideoMAEDecoder.__init__FTc s|rdnd}rdnd}t|jD]n\}} |r8||f}|jrh|jrhfdd} tjj| | |d} n| |dd} | d}r"|| df}q"|r||f}|dkr|dd| df}||}||} |st dd| ||fDSt | ||d S) Nr'csfdd}|S)Ncs|fSr:r'rrr'r(rszNVideoMAEDecoder.forward..create_custom_forward..custom_forwardr'rrrr(rsz6VideoMAEDecoder.forward..create_custom_forward)rrrrcss|]}|dk r|VqdSr:r'rr'r'r(rsz*VideoMAEDecoder.forward..)rrr) rrrrr$rrrrrr) rDrZreturn_token_numrrrrrrrrrrr'rr(rUs2        zVideoMAEDecoder.forward)FFT)r r!r"r<rUrWr'r'rEr(rs  rzXThe VideoMAE Model transformer with the decoder on top for self-supervised pre-training.c sjeZdZfddZeeeeedde j e j e e j e ee ee eeeefdddZZS) VideoMAEForPreTrainingcs~t|||_t||_tj|j|jdd|_ t t dd|j|_ t|jjj|j|_t||jjjd|_|dS)NFror)r?)r;r<rBrrrrur@rencoder_to_decoderryr$rz mask_tokenr8rQr?rArdecoderrrCrEr'r(r<s  zVideoMAEForPreTraining.__init__rNrc" CsF|dk r |n|jj}|j||||||d}|d}||}|j\} } } |dkrXtd|j| dd|} | |j  } | | | d| } | | | d| }tj|| |j|gdd}|||jd}|j}d}t|jjdkr|}nV|j }tt |ddddddf}tt |ddddddf}|||}|j\} }} }}|jj|jj}}|jjrF|| |||| ||||||}|ddd d d d d d}|| |||||||||| }||jddd|jdddd d}|| |||||||||| }n|jjdkr\td|| |||| ||||||}|ddd d d d d d}|| |||||||||| }|j\} }} || | d| }W5QRXt!} | ||}|s2|f|dd}!|dk r.|f|!S|!St"|||j#|j$dS)a bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, sequence_length)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Each video in the batch must have the same number of masked patches. Sequence length is `(num_frames // tubelet_size) * (image_size // patch_size) ** 2`. Returns: Examples: ```python >>> from transformers import AutoImageProcessor, VideoMAEForPreTraining >>> import numpy as np >>> import torch >>> num_frames = 16 >>> video = list(np.random.randint(0, 256, (num_frames, 3, 224, 224))) >>> image_processor = AutoImageProcessor.from_pretrained("MCG-NJU/videomae-base") >>> model = VideoMAEForPreTraining.from_pretrained("MCG-NJU/videomae-base") >>> pixel_values = image_processor(video, return_tensors="pt").pixel_values >>> num_patches_per_frame = (model.config.image_size // model.config.patch_size) ** 2 >>> seq_length = (num_frames // model.config.tubelet_size) * num_patches_per_frame >>> bool_masked_pos = torch.randint(0, 2, (1, seq_length)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss = outputs.loss ```N)rPrrrrrz!One must provided a boolean mask rGrrr rdr+rT)rkeepdim)rZunbiasedrgư>zQCan't unnormalize non-RGB images. Consider setting config.norm_pix_loss to False.r*rrr)%rBrrrrMrerAexpandrHrIrJrKrLrNr$catrrrZno_gradrTZ as_tensorrrr[rYZ norm_pix_lossrrfrrvarrr r)rr)"rDrOrPrrrrrrrRZseq_lenrTZexpanded_position_embeddingsZpos_emb_visibleZ pos_emb_maskZx_fullZdecoder_outputsrr*framesrJrrtimerirjr[rYZ frames_normZ videos_patchrSlabelsloss_fctrr'r'r(rUs(    ""       zVideoMAEForPreTraining.forward)NNNN)r r!r"r<rrrr)rr$r%rrrrrrrUrWr'r'rEr(rs    rzVideoMAE Model transformer with a video classification head on top (a linear layer on top of the average pooled hidden states of all tokens) e.g. for ImageNet.c sreZdZfddZeeeeedde e j e e j e e j e e e e e e e eefdddZZS) VideoMAEForVideoClassificationcsft||j|_t||_|jr0t|jnd|_ |jdkrPt |j|jnt |_ | dS)Nr)r;r< num_labelsrrrrrr@fc_normrur classifierrrCrEr'r(r<s   $z'VideoMAEForVideoClassification.__init__rN)rOrrrrrrmc Cs|dk r |n|jj}|j|||||d}|d}|jdk rL||d}n|dddf}||} d} |dk rT|jjdkr|jdkrd|j_n4|jdkr|jt j ks|jt j krd|j_nd|j_|jjdkrt } |jdkr| | | } n | | |} nN|jjdkr6t} | | d|j|d} n|jjdkrTt} | | |} |s| f|dd} | dk r| f| S| St| | |j|jd S) a3 labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Returns: Examples: ```python >>> import av >>> import torch >>> import numpy as np >>> from transformers import AutoImageProcessor, VideoMAEForVideoClassification >>> from huggingface_hub import hf_hub_download >>> np.random.seed(0) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`List[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`List[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # video clip consists of 300 frames (10 seconds at 30 FPS) >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample 16 frames >>> indices = sample_frame_indices(clip_len=16, frame_sample_rate=1, seg_len=container.streams.video[0].frames) >>> video = read_video_pyav(container, indices) >>> image_processor = AutoImageProcessor.from_pretrained("MCG-NJU/videomae-base-finetuned-kinetics") >>> model = VideoMAEForVideoClassification.from_pretrained("MCG-NJU/videomae-base-finetuned-kinetics") >>> inputs = image_processor(list(video), return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) ... logits = outputs.logits >>> # model predicts one of the 400 Kinetics-400 classes >>> predicted_label = logits.argmax(-1).item() >>> print(model.config.id2label[predicted_label]) eating spaghetti ```NrrrZ regressionZsingle_label_classificationZmulti_label_classificationrGr)rBrrrrrZ problem_typerZdtyper$longr`r Zsqueezer rr rrr) rDrOrrrrrrrrr*rrr'r'r(rUsR_      "    z&VideoMAEForVideoClassification.forward)NNNNNN)r r!r"r<rrrrrrr$rrrrrUrWr'r'rEr(rs$   r)Er#collections.abcr]rcopyr dataclassesrtypingrrrrnumpyr,r$Ztorch.utils.checkpointrZtorch.nnr r r Z activationsr Zmodeling_outputsrrZmodeling_utilsrZ pytorch_utilsrrrrrrrrZutils.constantsrrZconfiguration_videomaerZ get_loggerr loggerrZ_CHECKPOINT_FOR_DOCZ&VIDEOMAE_PRETRAINED_MODEL_ARCHIVE_LISTrr)r8Moduler9r=rkrrrrrrrZVIDEOMAE_START_DOCSTRINGrrrrrr'r'r'r(sr        5G(+9 J4