U 9%e(5@s4dZddlZddlZddlmZmZmZddlZddl Z ddl m m Z ddlZ ddl m Z ddlmZddlmZddlmZdd lmZmZmZmZmZmZdd lmZdd lmZm Z m!Z!m"Z"d d l#m$Z$e"%e&Z'dZ(dZ)dZ*d ddgZ+dZ,dZ-dZ.ddgZ/dZ0dZ1dddgZ2dWee3e3fe4e3ee j5e3ej6dddZ7Gddde j8Z9Gd d!d!e j8Z:Gd"d#d#e j8Z;Gd$d%d%e j8ZGd*d+d+e>Z?Gd,d-d-e j8Z@Gd.d/d/e j8ZAGd0d1d1e j8ZBGd2d3d3e j8ZCGd4d5d5e j8ZDGd6d7d7e j8ZEGd8d9d9e j8ZFGd:d;d;e j8ZGGdd?d?e j8ZIGd@dAdAeZJdBZKdCZLe dDeKGdEdFdFeJZMe dGeKGdHdIdIeJZNe dJeKGdKdLdLeJZOe dMeKGdNdOdOeJZPGdPdQdQe j8ZQGdRdSdSe j8ZRe dTeKGdUdVdVeJZSdS)Xz PyTorch WavLM model.N)OptionalTupleUnion)nn)CrossEntropyLoss)ACT2FN)is_deepspeed_zero3_enabled)BaseModelOutputCausalLMOutputSequenceClassifierOutputTokenClassifierOutputWav2Vec2BaseModelOutput XVectorOutput)PreTrainedModel)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardlogging) WavLMConfigrz1patrickvonplaten/wavlm-libri-clean-100h-base-plusi$izZ'mister quilter is the aposle of the middle classes and we are glad to welcome his gospel'gQ)@zmicrosoft/wavlm-base-plus-sdzmicrosoft/wavlm-base-plus-svg ףp= ?zmicrosoft/wavlm-basezmicrosoft/wavlm-base-pluszmicrosoft/wavlm-large)shape mask_prob mask_lengthattention_mask min_masksreturncs|\}dkrtdkr6tdddtjdfdd}|dk rt|d nfd d t|D}tj |ft d }g} |} | d kr|S|D]v} || } tjj t | d| dd} t | d krd}n| d }t| tj| | tjd |g} | | qt| } t| dddddf|| f} | || } t ddddf}t||| f|| }| |} | dkr҈d| | dk<t|| dd |S)af Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://arxiv.org/abs/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. rz&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `csXt|}t|}|kr2}|d|krTt|dd}|S)z;Given input length, compute how many spans should be maskedrr)intmax) input_lengthnum_masked_spanepsilonrrrsequence_lengthg/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/transformers/models/wavlm/modeling_wavlm.pycompute_num_masked_spanrs  z6_compute_mask_indices..compute_num_masked_spanNcsg|]}qSr&r&.0_)r%r&r' sz)_compute_mask_indices..dtyperF)replace) ValueErrornprandomranditemsumdetachtolistrangezerosboolchoicearangelenZ concatenateonesZint32appendarray broadcast_toZreshaper Zput_along_axis)rrrrr batch_sizer( input_lengthsZ spec_aug_maskZspec_aug_mask_idxsZmax_num_masked_spanr!r"Zspec_aug_mask_idxZdummy_mask_idxoffsetsr&r#r'_compute_mask_indicesLs`    rFcs&eZdZdfdd ZddZZS)WavLMNoLayerNormConvLayerrcsjt|dkr |j|dnd|_|j||_tj|j|j|j||j||j d|_ t |j |_ dS)Nrr kernel_sizestridebias)super__init__conv_dim in_conv_dim out_conv_dimrConv1d conv_kernel conv_stride conv_biasconvrfeat_extract_activation activationselfconfiglayer_id __class__r&r'rMs  z"WavLMNoLayerNormConvLayer.__init__cCs||}||}|SN)rUrWrY hidden_statesr&r&r'forwards  z!WavLMNoLayerNormConvLayer.forward)r__name__ __module__ __qualname__rMra __classcell__r&r&r\r'rGsrGcs&eZdZdfdd ZddZZS)WavLMLayerNormConvLayerrcs|t|dkr |j|dnd|_|j||_tj|j|j|j||j||j d|_ tj |jdd|_ t |j|_dS)NrrrHT)Zelementwise_affine)rLrMrNrOrPrrQrRrSrTrU LayerNorm layer_normrrVrWrXr\r&r'rMs  z WavLMLayerNormConvLayer.__init__cCs:||}|dd}||}|dd}||}|S)Nr))rU transposerirWr_r&r&r'ras      zWavLMLayerNormConvLayer.forward)rrbr&r&r\r'rgsrgcs&eZdZdfdd ZddZZS)WavLMGroupNormConvLayerrcst|dkr |j|dnd|_|j||_tj|j|j|j||j||j d|_ t |j |_ tj|j|jdd|_dS)NrrrHT) num_groupsZ num_channelsZaffine)rLrMrNrOrPrrQrRrSrTrUrrVrW GroupNormrirXr\r&r'rMs   z WavLMGroupNormConvLayer.__init__cCs"||}||}||}|Sr^)rUrirWr_r&r&r'ras   zWavLMGroupNormConvLayer.forward)rrbr&r&r\r'rlsrlcs$eZdZfddZddZZS)WavLMPositionalConvEmbeddingc sttj|j|j|j|jd|jd|_tjj }t tjj drNtjj j }t rddl }|jj|jjdd||jddd|_W5QRX|j||jj|j||jjn||jddd|_t|j|_t|j|_dS)Nr)rIpaddinggroups weight_normr)Z modifier_rankweight)namedim)rLrMrrQ hidden_sizenum_conv_pos_embeddingsZnum_conv_pos_embedding_groupsrUutilsrrhasattrZparametrizationsr deepspeedzeroZGatheredParametersrsZregister_external_parameterZweight_vZweight_gWavLMSamePadLayerrprrVrW)rYrZrrrzr\r&r'rMs(   z%WavLMPositionalConvEmbedding.__init__cCs:|dd}||}||}||}|dd}|SNrr)rkrUrprWr_r&r&r'ra+s      z$WavLMPositionalConvEmbedding.forwardrbr&r&r\r'ros rocs$eZdZfddZddZZS)r|cs$t|ddkrdnd|_dSNrrr)rLrMnum_pad_remove)rYrwr\r&r'rM8s zWavLMSamePadLayer.__init__cCs,|jdkr(|ddddd|j f}|S)Nr)rr_r&r&r'ra<s zWavLMSamePadLayer.forwardrbr&r&r\r'r|7s r|cs0eZdZdZfddZddZddZZS)WavLMFeatureEncoderz.Construct the features from raw audio waveformcstjdkr@tddgfddtjdD}n6jdkrdfddtjD}ntd jd t||_ d |_ d |_ dS) Ngrouprr[csg|]}t|ddqS)rr)rGr+irZr&r'r-Jsz0WavLMFeatureEncoder.__init__..rlayercsg|]}t|dqS)r)rgrrr&r'r-Nsz`config.feat_extract_norm` is z), but has to be one of ['group', 'layer']FT) rLrMZfeat_extract_normrlr9Znum_feat_extract_layersr1r ModuleList conv_layersgradient_checkpointing_requires_grad)rYrZrr\rr'rMFs      zWavLMFeatureEncoder.__init__cCs|D] }d|_qd|_dS)NF) parameters requires_gradrrYparamr&r&r'_freeze_parametersWs z&WavLMFeatureEncoder._freeze_parameterscCsj|dddf}|jr"|jr"d|_|jD]<}|jr\|jr\|jr\dd}tjj|||}q(||}q(|S)NTcsfdd}|S)Ncs|Sr^r&inputsmoduler&r'custom_forwardgszRWavLMFeatureEncoder.forward..create_custom_forward..custom_forwardr&rrr&rr'create_custom_forwardfs z:WavLMFeatureEncoder.forward..create_custom_forward)rtrainingrrrtorchrx checkpoint)rY input_valuesr`Z conv_layerrr&r&r'ra\s   zWavLMFeatureEncoder.forward)rcrdre__doc__rMrrarfr&r&r\r'rCs rcseZdZfddZZS)WavLMFeatureExtractorcs8t|td|jjd|jjdjdtdS)Nz The class `zD` has been depreciated and will be removed in Transformers v5. Use `rz ` instead.)rLrMwarningswarnr]rc __bases__ FutureWarningrYrZr\r&r'rMws  zWavLMFeatureExtractor.__init__)rcrdrerMrfr&r&r\r'rvsrcs$eZdZfddZddZZS)WavLMFeatureProjectioncsJttj|jd|jd|_t|jd|j|_ t |j |_ dS)Nr)Zeps) rLrMrrhrNlayer_norm_epsriLinearrv projectionDropoutZfeat_proj_dropoutdropoutrr\r&r'rMs zWavLMFeatureProjection.__init__cCs&||}||}||}||fSr^)rirr)rYr`Znorm_hidden_statesr&r&r'ras   zWavLMFeatureProjection.forwardrbr&r&r\r'rs rc seZdZdZdeeeeeedfdd Zdej e ej e ej ee ej e ej e e ej fd d dZ ej eejejfej eej ej fdddZeeej dddZej ej dddZZS)WavLMAttentionz=Multi-headed attention from 'Attention Is All You Need' paper@ T embed_dim num_headsr num_buckets max_distancehas_relative_position_biascst||_||_||_|||_|j||jkrNtd|jd|d|jd|_t |||_ t |||_ t |||_ t |||_ ||_||_ttd|jdd|_t |jd|_|rt|j|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gr)rLrMrrrZhead_dimr1Zscalingrrk_projv_projq_projout_projrr Parameterrr?gru_rel_pos_constgru_rel_pos_linearZ Embeddingrel_attn_embed)rYrrrrrrr\r&r'rMs(   zWavLMAttention.__init__NFr)r`r position_biasoutput_attentionsrcCs|\}}}|dkrH|||}|d|ddd||j||}||jdd|jdf} | dddd} || } | | jddd d} t | j ddd\} } | | |j d d } | ||jdd|}|d||f}|||||\}}|||fS) z'Attention layer with relative attentionNrrr)rr)rru?g@)size compute_bias unsqueezerepeatviewrrpermuterr6rZsigmoidchunkrtorch_multi_head_self_attention)rYr`rrrindexZbszZtgt_lenr,Zgated_hidden_statesZrelative_position_projZgate_aZgate_bZ gate_outputgated_position_bias attn_output attn_weightsr&r&r'ras(  $ zWavLMAttention.forward)r`rrrrcCs|dd}}}|dk r&|dnd}d} } d} tj||||j|jtdgt|j j |j j |j j f| | | |j |jj|jj |j|||d|j j|j j|j jd\} } | dd} | dk r| dddf| jdd|jf| jdd} | | fS)zCsimple wrapper around torch's multi_head_attention_forward functionrrNFT)Zuse_separate_proj_weightZ q_proj_weightZ k_proj_weightZ v_proj_weight)rkneFZmulti_head_attention_forwardrrremptycatrrKrrrrrsrrBr)rYr`rrrquerykeyvalueZkey_padding_maskZbias_kZbias_vZ add_zero_attnrrr&r&r'rsB    "z.WavLMAttention.torch_multi_head_self_attention) query_length key_lengthrcCsxtj|tjddddf}tj|tjddddf}||}||}||jjj}||}|dddg}|S)Nr.rrr) rr=long_relative_positions_buckettorrsdevicer)rYrrZcontext_positionZmemory_positionZrelative_positionZrelative_position_bucketvaluesr&r&r'rs  zWavLMAttention.compute_bias)relative_positionsrcCs|jd}|dktj|}t|}|d}||k}t||}|t|j|}|||}||tj}t |t ||d}|t |||7}|Sr~) rrrrabslogfloatmathrminZ full_likewhere)rYrrZrelative_bucketsZ max_exactZis_smallZrelative_positions_if_largeZrelative_position_if_larger&r&r'r"s   z)WavLMAttention._relative_positions_bucket)rrrT)NNFr)rcrdrerrrr;rMrTensorrrra FloatTensorr LongTensorZ BoolTensorrrrrfr&r&r\r'rs@' + 7 rcs$eZdZfddZddZZS)WavLMFeedForwardcsptt|j|_t|j|j|_ t |j t rDt |j |_n|j |_t|j|j|_t|j|_dSr^)rLrMrrZactivation_dropoutintermediate_dropoutrrvZintermediate_sizeintermediate_dense isinstanceZ hidden_actstrrintermediate_act_fn output_densehidden_dropoutoutput_dropoutrr\r&r'rM9s  zWavLMFeedForward.__init__cCs6||}||}||}||}||}|Sr^)rrrrrr_r&r&r'raFs      zWavLMFeedForward.forwardrbr&r&r\r'r8s rcs0eZdZd eedfdd Zd dd ZZS) WavLMEncoderLayerTrZrcsntt|j|j|j|j|j|d|_t |j |_ t j |j|jd|_t||_t j |j|jd|_dSNrrrLrMrrvZnum_attention_headsZattention_dropoutrZmax_bucket_distance attentionrrrrrhrrir feed_forwardfinal_layer_normrYrZrr\r&r'rMQs  zWavLMEncoderLayer.__init__NFrc Csl|}|j|||||d\}}}||}||}||}|||}||}||f}|rh||f7}|S)Nrrrr)rrrirr) rYr`rrrr attn_residualroutputsr&r&r'ra`s"     zWavLMEncoderLayer.forward)T)NNFrrcrdrerr;rMrarfr&r&r\r'rPsrcs0eZdZd eedfdd Zd ddZZS) WavLMEncoderLayerStableLayerNormTrcsntt|j|j|j|j|j|d|_t |j |_ t j |j|jd|_t||_t j |j|jd|_dSrrrr\r&r'rMzs  z)WavLMEncoderLayerStableLayerNorm.__init__NFcCsf|}||}|j||||d\}}}||}||}||||}||f}|rb||f7}|S)N)rrr)rirrrr)rYr`rrrrrrr&r&r'ras    z(WavLMEncoderLayerStableLayerNorm.forward)T)NNFrr&r&r\r'rysrcs&eZdZfddZdddZZS) WavLMEncodercsft|_t|_tjjjd|_ t j |_ t fddtjD|_d|_dS)Nrcsg|]}t|dkdqSr)r)rrrr&r'r-sz)WavLMEncoder.__init__..FrLrMrZropos_conv_embedrrhrvrrirrrrr9num_hidden_layerslayersrrr\rr'rMs  zWavLMEncoder.__init__NFTcsT|rdnd}rdnd}|dk r*d||<||}||}||}||}t} d} t|jD]\} } |rz||f}tg} |jo| dko| |j j k}|r| r|j r|jr؇fdd}tj j || ||| }n| ||| | d}|dd\}} |rd}rd||df}qd|r(||f}|sFtd d |||fDSt|||d S) Nr&rrcsfdd}|S)Ncs|fSr^r&rrrr&r'rszKWavLMEncoder.forward..create_custom_forward..custom_forwardr&rrrr'rsz3WavLMEncoder.forward..create_custom_forwardrrNNcss|]}|dk r|VqdSr^r&r+vr&r&r' sz'WavLMEncoder.forward..last_hidden_stater` attentions)rrirr enumeraterrr4rrZ layerdroprrxrtupler rYr`rroutput_hidden_states return_dictZall_hidden_statesZall_self_attentionsZposition_embeddingsZdeepspeed_zero3_is_enabledrrrZdropout_probabilityZskip_the_layerrZ layer_outputsr&rr'rasZ           zWavLMEncoder.forward)NFFTrbr&r&r\r'rs  rcs&eZdZfddZdddZZS) WavLMEncoderStableLayerNormcsft|_t|_tjjjd|_ t j |_ t fddtjD|_d|_dS)Nrcsg|]}t|dkdqSr)rrrr&r'r-sz8WavLMEncoderStableLayerNorm.__init__..Frrr\rr'rMs   z$WavLMEncoderStableLayerNorm.__init__NFTcsP|rdnd}rdnd}|dk r*d||<||}||}||}t} d} t|jD]\} } |rp||f}tg} |jo| dko| |jj k}|r| r|j r|jr·fdd}tj j || ||| }n| ||| d}|dd\}} |rd}rZ||df}qZ| |}|r$||f}|sBtdd |||fDSt|||d S) Nr&rcsfdd}|S)Ncs|fSr^r&rrr&r'r)szZWavLMEncoderStableLayerNorm.forward..create_custom_forward..custom_forwardr&rrrr'r(szBWavLMEncoderStableLayerNorm.forward..create_custom_forward)rrrrrcss|]}|dk r|VqdSr^r&rr&r&r'rIsz6WavLMEncoderStableLayerNorm.forward..r)rrr r rrr4rrZr rrxrrir r r r&rr'rasX           z#WavLMEncoderStableLayerNorm.forward)NFFTrbr&r&r\r'rs  rcs4eZdZdZfddZeddZddZZS)WavLMGumbelVectorQuantizerz Vector quantization using gumbel softmax. See [CATEGORICAL REPARAMETERIZATION WITH GUMBEL-SOFTMAX](https://arxiv.org/pdf/1611.01144.pdf) for more information. cst|j|_|j|_|j|jdkrDtd|jd|jdt t d|j|j|j|j|_ t |jd|j|j|_d|_dS)Nrz`config.codevector_dim z5 must be divisible by `config.num_codevector_groups` z for concatenation.rr)r)rLrMZnum_codevector_groupsrmZnum_codevectors_per_groupnum_varsZcodevector_dimr1rrrr codevectorsrrN weight_proj temperaturerr\r&r'rMUs z#WavLMGumbelVectorQuantizer.__init__cCs8|jdd}ttj|t|ddd }|S)NrrgHz>r))meanrexpr6r)ZprobsZmarginal_probs perplexityr&r&r'_compute_perplexityjs (z.WavLMGumbelVectorQuantizer._compute_perplexityc Cs|j\}}}||}||||jd}|jrtjj||j dd}| |}t j ||||jddd}| |}nH|jdd}|j|jd|ddd}||||jd}| |}|||d}|d|j} | |||j|jd} | d||d} | |fS)Nr)T)tauhardrrrrj)rrrrmrr functionalZgumbel_softmaxrrZtype_asrsoftmaxrargmaxZ new_zerosZscatter_rrrr6) rYr`rCr%rvZcodevector_probsZcodevector_soft_distrZcodevector_idxZcodevectors_per_grouprr&r&r'raps0        z"WavLMGumbelVectorQuantizer.forward) rcrdrerrM staticmethodrrarfr&r&r\r'rOs   rcs$eZdZfddZddZZS) WavLMAdaptercsptjjkr8tjj|_tj|_n d|_|_t fddt j D|_ j |_ dS)Nc3s|]}tVqdSr^)WavLMAdapterLayerr*rr&r'rsz(WavLMAdapter.__init__..)rLrMoutput_hidden_sizervrrprojrhproj_layer_normrr9num_adapter_layersrr rr\rr'rMs    zWavLMAdapter.__init__cCsr|jdk r(|jdk r(||}||}|dd}|jD]&}tj}|jrX||jkr:||}q:|dd}|Sr})r#r$rkrr2r3rr )rYr`rZlayerdrop_probr&r&r'ras       zWavLMAdapter.forwardrbr&r&r\r'r s r cs$eZdZfddZddZZS)r!cs0ttj|jd|j|j|jdd|_dS)Nrr)rJrp)rLrMrrQr"Zadapter_kernel_sizeadapter_striderUrr\r&r'rMs zWavLMAdapterLayer.__init__cCs||}tjj|dd}|S)Nrr)rUrrZglur_r&r&r'ras zWavLMAdapterLayer.forwardrbr&r&r\r'r!s r!c@sfeZdZdZeZdZdZdZddZ de e j e feedd d Zde e j d d d ZdddZdS)WavLMPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. wavlmrTc Cst|tr>|jjjjddd|jjjtj |j njt|t rtj j|j jddtd|j jd|j jdtj |j jdnt|trtd|jj}tj j |jj| |dtj j |jj| |dnt|tjr|jjjd|jjd|jdk r|jjnt|tjtjfrN|jj|jjdnZt|tjrtj |j|jdk rt|j|j|jd}tj j |j| |ddS) zInitialize the weightsrr)rstdrr)abNr)rrrrsdataZnormal_rKZzero_rinituniform_rrorUrsqrtrIZ in_channelsZ constant_rrZ in_featuresrrZZinitializer_rangerhrnZfill_rQZkaiming_normal_rq)rYrkr&r&r' _init_weightss6       z"WavLMPreTrainedModel._init_weightsN)rD add_adaptercCsn|dkr|jjn|}dd}t|jj|jjD]\}}||||}q.|rjt|jjD]}||d|jj}qT|S)zH Computes the output length of the convolutional layers NcSstj|||dddS)Nfloor)Z rounding_moder)rdivr!rIrJr&r&r'_conv_out_lengthszOWavLMPreTrainedModel._get_feat_extract_output_lengths.._conv_out_lengthr)rZr2ziprRrSr9r%r&)rYrDr2r6rIrJr,r&r&r' _get_feat_extract_output_lengthssz5WavLMPreTrainedModel._get_feat_extract_output_lengths)feature_vector_lengthrcCs|jdddddf}|j||d}|tj}|jd}tj||f|j|jd}d|tj |jd|jd|df<| dgd dg }|S)Nr)rr2r)r/rr)r) Zcumsumr8rrrrr:r/rr=flipr;)rYr9rr2Znon_padded_lengthsZoutput_lengthsrCr&r&r'"_get_feature_vector_attention_mask s  "z7WavLMPreTrainedModel._get_feature_vector_attention_maskFcCst|tttfr||_dSr^)rrrrr)rYrrr&r&r'_set_gradient_checkpointingsz0WavLMPreTrainedModel._set_gradient_checkpointing)N)N)F)rcrdrerr config_classZbase_model_prefixZmain_input_nameZsupports_gradient_checkpointingr1rrrrrr;r8r<r=r&r&r&r'r's""   r'a WavLM was proposed in [WavLM: Unified Speech Representation Learning with Labeled and Unlabeled Data](https://arxiv.org/abs/2110.13900) by Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, Furu Wei. This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving etc.). This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`WavLMConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. aI Args: input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a `.flac` or `.wav` audio file into an array of type `List[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoProcessor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`Wav2Vec2Processor.__call__`] for details. attention_mask (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing convolution and attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) `attention_mask` should only be passed if the corresponding processor has `config.return_attention_mask == True`. For all models whose processor has `config.return_attention_mask == False`, `attention_mask` should **not** be passed to avoid degraded performance when doing batched inference. For such models `input_values` should simply be padded with 0 and passed without `attention_mask`. Be aware that these models also yield slightly different results depending on whether `input_values` is padded or not. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z_The bare WavLM Model transformer outputting raw hidden-states without any specific head on top.c seZdZedfdd ZddZddZdeje eje ej d d d Z e e eeeed ed de eje eje eje ee ee eeeefdddZZS) WavLMModelrcst|||_t||_t||_|jdks:|jdkrRt t |j |_|jrdt||_n t||_|jr|t|nd|_|dS)Nr)rLrMrZrfeature_extractorrfeature_projectionmask_time_probmask_feature_probrrrrrvr.masked_spec_embedZdo_stable_layer_normrencoderrr2r adapter post_initrr\r&r'rMcs     zWavLMModel.__init__cCstdt|dSz Calling this function will disable the gradient computation for the feature encoder so that its parameters will not be updated during training. The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5.Please use the equivalent `freeze_feature_encoder` method instead.Nrrrfreeze_feature_encoderrYr&r&r'freeze_feature_extractorws z#WavLMModel.freeze_feature_extractorcCs|jdS Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training. N)r@rrLr&r&r'rKsz!WavLMModel.freeze_feature_encoderN)r`mask_time_indicesrcCst|jdds|S|\}}}|dk r<|j|j||<nZ|jjdkr|jrt||f|jj|jj ||jj d}t j ||j t jd}|j|j||<|jjdkr|jrt||f|jj|jj|jjd}t j ||j t jd}|dddfd|d}d||<|S) z Masks extracted features along time axis and/or along feature axis according to [SpecAugment](https://arxiv.org/abs/1904.08779). Zapply_spec_augmentTNr)rrrr)rr/)rrrr))getattrrZrrDrr/rBrrFZmask_time_lengthZmask_time_min_masksrZtensorrr;rCZmask_feature_lengthZmask_feature_min_masksexpand)rYr`rPrrCr%rvZmask_feature_indicesr&r&r'_mask_hidden_statess4 zWavLMModel._mask_hidden_statesaudior output_typer>modalityexpected_output)rrrPrrrrc Cs|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}||}|dd}|dk rp|j|jd|dd}||\}}|j |||d}|j |||||d} | d}|j dk r| |}|s||f| ddSt ||| j | jdS) NrrFr:)rPrrrrrr)rextract_featuresr`r )rZrruse_return_dictr@rkr<rrArSrErFrr`r ) rYrrrPrrrrZr`Zencoder_outputsr&r&r'rasH    zWavLMModel.forward)NN)NNNNN)rcrdrerrMrMrKrrrrrSrWAVLM_INPUTS_DOCSTRINGr_CHECKPOINT_FOR_DOCr_CONFIG_FOR_DOC_EXPECTED_OUTPUT_SHAPErr;rrrarfr&r&r\r'r?]s@   .  r?zcWavLM Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC).c seZdZdeedfdd ZddZddZd d Zd d Z e e e e eeeed deejeejeeeeeeeejeeefdddZZS) WavLMForCTCN) target_langcs~t|t||_t|j|_||_|j dkrFt d|j dt |dr\|j r\|jn|j}t||j |_|dS)NzYou are trying to instantiate z with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `WavLMForCTC.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration.r2)rLrMr?r(rrZ final_dropoutrra vocab_sizer1r]ryr2r"rvrlm_headrG)rYrZrar"r\r&r'rMs    zWavLMForCTC.__init__cCsr|j}|dk r2t|jdddkr2td|dn<|dkrXt|jdddk rXtdn|dk rn|j|dddS)a' This method overwrites [`~PreTrainedModel.tie_weights`] so that adapter weights can be correctly loaded when passing `target_lang=...` to `from_pretrained(...)`. This method is **not** supposed to be called by the user and is prone to be changed in the future. NZadapter_attn_dimzCannot pass `target_lang`: z- if `config.adapter_attn_dim` is not defined.z)By default `target_lang` is set to 'eng'.T)Z force_load)rarQrZr1loggerinfoZ load_adapter)rYrar&r&r' tie_weightss  zWavLMForCTC.tie_weightscCstdt|dSrOrINrJrLr&r&r'rM's z$WavLMForCTC.freeze_feature_extractorcCs|jjdSrNr(r@rrLr&r&r'rK3sz"WavLMForCTC.freeze_feature_encodercCs|jD] }d|_q dSz Calling this function will disable the gradient computation for the base model so that its parameters will not be updated during training. Only the classification head will be updated. FNr(rrrr&r&r'freeze_base_model:szWavLMForCTC.freeze_base_model)rrVr>rXZ expected_lossrrrrrlabelsrc Csf|dk r |n|jj}|j|||||d}|d}||}||} d} |dk r"||jjkrttd|jj|dk r|ntj |tj d}| | d tj } |dk} | d} || }tjj| dtjddd}tjjjd d ,tjj||| | |jj|jj|jjd } W5QRX|sR| f|td}| dk rN| f|S|St| | |j|jd S) a labels (`torch.LongTensor` of shape `(batch_size, target_length)`, *optional*): Labels for connectionist temporal classification. Note that `target_length` has to be smaller or equal to the sequence length of the output logits. Indices are selected in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size - 1]`. NrYrz$Label values must be <= vocab_size: r.r))rur/rF)enabled)blankZ reductionZ zero_infinitylosslogitsr`r )rZr[r(rrcr rbr1rZ ones_likerr8r6rZ masked_selectrrZ log_softmaxZfloat32rkbackendsZcudnnflagsZctc_lossZ pad_token_idZctc_loss_reductionZctc_zero_infinity_HIDDEN_STATES_START_POSITIONr r`r )rYrrrrrrmrr`rrrqrDZ labels_maskZtarget_lengthsZflattened_targetsZ log_probsoutputr&r&r'raBsR      zWavLMForCTC.forward)N)NNNNN)rcrdrerrrMrfrMrKrkrr\rr]r r^_CTC_EXPECTED_OUTPUT_CTC_EXPECTED_LOSSrrr;rrrarfr&r&r\r'r`s6   r`z WavLM Model with a sequence classification head on top (a linear layer over the pooled output) for tasks like SUPERB Keyword Spotting. c seZdZfddZddZddZddZeee e e e d d de eje eje ee ee ee ejeee fd d dZZS)WavLMForSequenceClassificationcst|t|dr$|jr$tdt||_|jd}|jrTt t |||_ t |j|j|_t |j|j|_|dS)Nr2z\Sequence classification does not support the use of WavLM adapters (config.add_adapter=True)r)rLrMryr2r1r?r(ruse_weighted_layer_sumrrrr? layer_weightsrrvZclassifier_proj_size projector num_labels classifierrGrYrZ num_layersr\r&r'rMs   z'WavLMForSequenceClassification.__init__cCstdt|dSrHrJrLr&r&r'rMs z7WavLMForSequenceClassification.freeze_feature_extractorcCs|jjdSrNrhrLr&r&r'rKsz5WavLMForSequenceClassification.freeze_feature_encodercCs|jD] }d|_q dSrirjrr&r&r'rksz0WavLMForSequenceClassification.freeze_base_modelrT)rrVr>rWNrlcCsf|dk r |n|jj}|jjr dn|}|j|||||d}|jjr|t}tj|dd}tjj |j dd} || dddj dd}n|d}| |}|dkr|jdd} n<||jd|} d|| <|j dd| j dd dd} || } d} |dk r"t}|| d|jj| d} |sR| f|td}| dk rN| f|S|St| | |j|jd S)  labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NTrYrrr)rrrp)rZr[rzr(rurstackrrrr{rr6r|rr<rr~rr}r r`r )rYrrrrrrmrr` norm_weightsZ pooled_outputZ padding_maskrrrqloss_fctrvr&r&r'rasF     z&WavLMForSequenceClassification.forward)NNNNN)rcrdrerMrMrKrkrr\rr]r r^rrrr;rrrarfr&r&r\r'rys2    ryza WavLM Model with a frame classification head on top for tasks like Speaker Diarization. c seZdZfddZddZddZddZeee e e e d e d deejeejeejeeeeeeeee fd d dZZS) WavLMForAudioFrameClassificationcszt|t|dr$|jr$tdt||_|jd}|jrTt t |||_ t |j|j|_|j|_|dS)Nr2z_Audio frame classification does not support the use of WavLM adapters (config.add_adapter=True)r)rLrMryr2r1r?r(rrzrrrr?r{rrvr}r~ init_weightsrr\r&r'rMs   z)WavLMForAudioFrameClassification.__init__cCstdt|dSrgrJrLr&r&r'rM's z9WavLMForAudioFrameClassification.freeze_feature_extractorcCs|jjdSrNrhrLr&r&r'rK3sz7WavLMForAudioFrameClassification.freeze_feature_encodercCs|jD] }d|_q dSrirjrr&r&r'rk:sz2WavLMForAudioFrameClassification.freeze_base_modelrTrUN)rrrmrrrrcCs|dk r |n|jj}|jjr dn|}|j|||||d}|jjr|t}tj|dd}tjj |j dd} || dddj dd}n|d}| |} d} |dk rt} | | d|jtj| d|jdd} |s| f|td} | St| | |j|jd S) rNTrYrrr)r)Zaxisrp)rZr[rzr(rurrrrrr{rr6r~rr}rr r`r )rYrrrmrrrrr`rrrrqrrvr&r&r'raBs: (z(WavLMForAudioFrameClassification.forward)NNNNN)rcrdrerMrMrKrkrr\r_FRAME_CLASS_CHECKPOINTr r^_FRAME_EXPECTED_OUTPUTrrrr;rrrarfr&r&r\r'rs4    rcs&eZdZdfdd ZddZZS) AMSoftmaxLoss>@皙?csFtt|||_||_||_tjt ||dd|_ t |_ dS)NT)r) rLrrMscalemarginr}rrrZrandnrsrrq)rYZ input_dimr}rrr\r&r'rMs zAMSoftmaxLoss.__init__c Csx|}tjj|jdd}tjj|dd}t||}||j}tj||j }|j t | ||}| ||}|S)Nrrr)flattenrr normalizersrmmrZone_hotr}rrr;rq) rYr`rmrsZ cos_thetapsiZonehotrrrqr&r&r'ras   zAMSoftmaxLoss.forward)rrrbr&r&r\r'rsrcs&eZdZdfdd ZddZZS) TDNNLayerrcsvt|dkr |j|dn|j||_|j||_|j||_|j||_t |j|j|j|_ t |_ dS)Nrr)rLrMtdnn_dimrOrP tdnn_kernelrIZ tdnn_dilationdilationrrkernelZReLUrWrXr\r&r'rMs "   zTDNNLayer.__init__cCsV|d}tjj||j|jfd|jf|jdfd}|dd}||}| |}|S)Nr)rJrr) rrrZunfoldrIrOrrkrrWr_r&r&r'ras     zTDNNLayer.forward)rrbr&r&r\r'rs rzi WavLM Model with an XVector feature extraction head on top for tasks like Speaker Verification. c seZdZfddZddZddZddZeej e fd d d Z e e eeeed ed deejeejeeeeeeeejeeefdddZZS)WavLMForXVectorcstt|_jd}jr.r)r)rLrMr?r(rrzrrrr?r{rrvrr|r9r>rtdnnZxvector_output_dimr@r~rr} objectiver)rYrZrZ tdnn_layersr\rr'rMs    zWavLMForXVector.__init__cCstdt|dSrgrJrLr&r&r'rMs z(WavLMForXVector.freeze_feature_extractorcCs|jjdSrNrhrLr&r&r'rKsz&WavLMForXVector.freeze_feature_encodercCs|jD] }d|_q dSrirjrr&r&r'rksz!WavLMForXVector.freeze_base_model)rDcCs&dd}|jjD]}|||d}q|S)z? Computes the output length of the TDNN layers cSs|||dS)Nrr&r5r&r&r'r6szBWavLMForXVector._get_tdnn_output_lengths.._conv_out_lengthr)rZr)rYrDr6rIr&r&r'_get_tdnn_output_lengthss z(WavLMForXVector._get_tdnn_output_lengthsrTrUNrlcCs|dk r |n|jj}|jjr dn|}|j|||||d}|jjr|t}tj|dd}tjj |j dd} || dddj dd}n|d}| |}|jD] } | |}q|dkr|jdd} |jdd} n||j dd} || }g} g} t|D]D\}}| ||d|fjdd| ||d|fjddqt| } t| } tj| | gdd}||}||}d}|dk r|||}|s||f|td}|dk r|f|S|St||||j|jdS) rNTrYrrr)r)rqrrZ embeddingsr`r )rZr[rzr(rurrrrrr{rr6r|rrr)r8rr r@rr@r~rrr`r )rYrrrrrrmrr`rZ tdnn_layerZ mean_featuresZ std_featuresZfeat_extract_output_lengthsZtdnn_output_lengthsrlengthZstatistic_poolingZoutput_embeddingsrrrqrvr&r&r'ras\            zWavLMForXVector.forward)NNNNN)rcrdrerMrMrKrkrrrrrrr\r_XVECTOR_CHECKPOINTrr^_XVECTOR_EXPECTED_OUTPUTrrr;rrarfr&r&r\r'rs6    r)Nr)Trrrtypingrrrnumpyr2rZtorch.nn.functionalrrrZtorch.utils.checkpointZtorch.nnrZ activationsrZintegrations.deepspeedr Zmodeling_outputsr r r r rrZmodeling_utilsrrxrrrrZconfiguration_wavlmrZ get_loggerrcrdrur^r]r_rwrxrrrrZ#WAVLM_PRETRAINED_MODEL_ARCHIVE_LISTrrrZndarrayrFModulerGrgrlror|rrrrrrrrrrr r!r'ZWAVLM_START_DOCSTRINGr\r?r`ryrrrrr&r&r&r's            x( 3 ()%XYG \%vk