U ,-eO@sdZddlZddlZddlmZddlmZmZmZddl Z ddl Z ddl Z ddl m Z ddlmZddlmZdd lmZdd lmZmZmZmZmZmZmZdd lmZdd lmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&d dl'm(Z(dZ)dZ*e$rddl+m,Z-e%.e/Z0dZ1dZ2dZ3d ddgZ4dZ5dZ6dZ7dZ8dZ9dZ:ddgZ;dZeGd"d#d#eZ?dkee@e@feAe@ee jBe@e jCd$d%d&ZDdlee@ee jCd'd(d)ZEGd*d+d+e jFZGGd,d-d-e jFZHGd.d/d/e jFZIGd0d1d1e jFZJGd2d3d3e jFZKGd4d5d5e jFZLGd6d7d7eLZMGd8d9d9e jFZNGd:d;d;e jFZOGdd?d?e jFZQGd@dAdAe jFZRGdBdCdCe jFZSGdDdEdEe jFZTGdFdGdGe jFZUGdHdIdIe jFZVGdJdKdKe jFZWGdLdMdMe jFZXGdNdOdOeZYdPZZdQZ[e!dReZGdSdTdTeYZ\e!dUeZGdVdWdWeYZ]e!dXeZGdYdZdZeYZ^e!d[eZGd\d]d]eYZ_e!d^eZGd_d`d`eYZ`e!daeZGdbdcdceYZaGdddedee jFZbGdfdgdge jFZce!dheZGdidjdjeYZddS)mz PyTorch Wav2Vec2 model.N) dataclass)OptionalTupleUnion)nn)CrossEntropyLoss)ACT2FN)is_deepspeed_zero3_enabled)BaseModelOutputCausalLMOutputMaskedLMOutputSequenceClassifierOutputTokenClassifierOutputWav2Vec2BaseModelOutput XVectorOutput)PreTrainedModel) ModelOutputadd_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forward cached_fileis_safetensors_availableloggingreplace_return_docstrings)Wav2Vec2Configzadapter.{}.binzadapter.{}.safetensors) load_filerzfacebook/wav2vec2-base-960hi$iz['MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL'g= ףpJ@zsuperb/wav2vec2-base-superb-ksz '_unknown_'g)\(@zanton-l/wav2vec2-base-superb-sdzanton-l/wav2vec2-base-superb-svg\(\?zfacebook/wav2vec2-large-960hz!facebook/wav2vec2-large-960h-lv60z&facebook/wav2vec2-large-960h-lv60-selfc@seZdZUdZdZeejed<dZ ejed<dZ ejed<dZ ejed<dZ ee ejed<dZee ejed<dZeejed <dZeejed <dS) Wav2Vec2ForPreTrainingOutputa1 Output type of [`Wav2Vec2ForPreTraining`], with potential hidden states and attentions. Args: loss (*optional*, returned when `sample_negative_indices` are passed, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the contrastive loss (L_m) and the diversity loss (L_d) as stated in the [official paper](https://arxiv.org/pdf/2006.11477.pdf) . (classification) loss. projected_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`): Hidden-states of the model projected to *config.proj_codevector_dim* that can be used to predict the masked projected quantized states. projected_quantized_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`): Quantized extracted feature vectors projected to *config.proj_codevector_dim* representing the positive target vectors for contrastive loss. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. contrastive_loss (*optional*, returned when `sample_negative_indices` are passed, `torch.FloatTensor` of shape `(1,)`): The contrastive loss (L_m) as stated in the [official paper](https://arxiv.org/pdf/2006.11477.pdf) . diversity_loss (*optional*, returned when `sample_negative_indices` are passed, `torch.FloatTensor` of shape `(1,)`): The diversity loss (L_d) as stated in the [official paper](https://arxiv.org/pdf/2006.11477.pdf) . Nlossprojected_statesprojected_quantized_statescodevector_perplexity hidden_states attentionscontrastive_lossdiversity_loss)__name__ __module__ __qualname____doc__r rtorch FloatTensor__annotations__r!r"r#r$rr%r&r'r/r/o/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.pyrcs r)shape mask_prob mask_lengthattention_mask min_masksreturncs|\}dkrtdkr6tdddtjdfdd}|dk rt|d nfd d t|D}tj |ft d }g} |} | d kr|S|D]v} || } tjj t | d| dd} t | d krd}n| d }t| tj| | tjd |g} | | qt| } t| dddddf|| f} | || } t ddddf}t||| f|| }| |} | dkr҈d| | dk<t|| dd |S)af Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://arxiv.org/abs/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. rz&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `csXt|}t|}|kr2}|d|krTt|dd}|S)z;Given input length, compute how many spans should be maskedrr)intmax) input_lengthnum_masked_spanepsilonr3r2r5sequence_lengthr/r0compute_num_masked_spans  z6_compute_mask_indices..compute_num_masked_spanNcsg|]}qSr/r/.0_)r>r/r0 sz)_compute_mask_indices..dtyperF)replace) ValueErrornprandomranditemsumdetachtolistrangezerosboolchoicearangelenZ concatenateonesint32appendarray broadcast_toreshaper9Zput_along_axis)r1r2r3r4r5 batch_sizer? input_lengthsZ spec_aug_maskZspec_aug_mask_idxsZmax_num_masked_spanr:r;Zspec_aug_mask_idxZdummy_mask_idxoffsetsr/r<r0_compute_mask_indicess`    r_)features_shape num_negativesmask_time_indicesc Cs|\}}t|}tj|||ftjd}|dk r:|tn tj|td}t|D]}||d}|||} t t|ddddf|d|f} tj j d||d|fd} | | | kd7<| | ||||<||||7<qP|S)z> Sample `num_negatives` vectors from feature vectors. )r1rFNrErr)size) rIrTrQrWZastyperRrVrPrMrZrJrandint) r`rarbr\r>Zsequence_length_rangesampled_negative_indicesZ batch_idxhighZmapped_masked_indicesZfeature_indicesZsampled_indicesr/r/r0_sample_negative_indicess   *rgcs&eZdZdfdd ZddZZS)Wav2Vec2NoLayerNormConvLayerrcsjt|dkr |j|dnd|_|j||_tj|j|j|j||j||j d|_ t |j |_ dS)Nrr kernel_sizestridebias)super__init__conv_dim in_conv_dim out_conv_dimrConv1d conv_kernel conv_stride conv_biasconvr feat_extract_activation activationselfconfiglayer_id __class__r/r0rn)s  z%Wav2Vec2NoLayerNormConvLayer.__init__cCs||}||}|SN)rvrxrzr$r/r/r0forward7s  z$Wav2Vec2NoLayerNormConvLayer.forward)rr(r)r*rnr __classcell__r/r/r}r0rh(srhcs&eZdZdfdd ZddZZS)Wav2Vec2LayerNormConvLayerrcs|t|dkr |j|dnd|_|j||_tj|j|j|j||j||j d|_ tj |jdd|_ t |j|_dS)NrrriT)Zelementwise_affine)rmrnrorprqrrrrsrtrurv LayerNorm layer_normr rwrxryr}r/r0rn>s  z#Wav2Vec2LayerNormConvLayer.__init__cCs:||}|dd}||}|dd}||}|S)Nr@)rv transposerrxrr/r/r0rMs      z"Wav2Vec2LayerNormConvLayer.forward)rrr/r/r}r0r=srcs&eZdZdfdd ZddZZS)Wav2Vec2GroupNormConvLayerrcst|dkr |j|dnd|_|j||_tj|j|j|j||j||j d|_ t |j |_ tj|j|jdd|_dS)NrrriT) num_groupsZ num_channelsZaffine)rmrnrorprqrrrrsrtrurvr rwrx GroupNormrryr}r/r0rnYs   z#Wav2Vec2GroupNormConvLayer.__init__cCs"||}||}||}|Sr)rvrrxrr/r/r0ris   z"Wav2Vec2GroupNormConvLayer.forward)rrr/r/r}r0rXsrcs$eZdZfddZddZZS)Wav2Vec2PositionalConvEmbeddingc sttj|j|j|j|jd|jd|_tjj }t tjj drNtjj j }t rddl }|jj|jjdd||jddd|_W5QRX|j||jj|j||jjn||jddd|_t|j|_t|j|_dS)Nr)rjpaddinggroups weight_normr)Z modifier_rankweight)namedim)rmrnrrr hidden_sizenum_conv_pos_embeddingsZnum_conv_pos_embedding_groupsrvutilsrhasattrZparametrizationsr deepspeedzeroZGatheredParametersrZregister_external_parameterZweight_vZweight_gWav2Vec2SamePadLayerrr rwrx)rzr{rrr}r/r0rnqs(   z(Wav2Vec2PositionalConvEmbedding.__init__cCs:|dd}||}||}||}|dd}|SNrr)rrvrrxrr/r/r0rs      z'Wav2Vec2PositionalConvEmbedding.forwardrr/r/r}r0rps rcs$eZdZfddZddZZS)rcs$t|ddkrdnd|_dS)Nrrr)rmrnnum_pad_remove)rzrr}r/r0rns zWav2Vec2SamePadLayer.__init__cCs,|jdkr(|ddddd|j f}|S)Nr)rrr/r/r0rs zWav2Vec2SamePadLayer.forwardrr/r/r}r0rs rcs0eZdZdZfddZddZddZZS)Wav2Vec2FeatureEncoderz.Construct the features from raw audio waveformcstjdkr@tddgfddtjdD}n6jdkrdfddtjD}ntd jd t||_ d |_ d |_ dS) Ngrouprr|csg|]}t|ddqS)rr)rhrBir{r/r0rDsz3Wav2Vec2FeatureEncoder.__init__..rlayercsg|]}t|dqS)r)rrrr/r0rDsz`config.feat_extract_norm` is z), but has to be one of ['group', 'layer']FT) rmrnZfeat_extract_normrrPZnum_feat_extract_layersrHr ModuleList conv_layersgradient_checkpointing_requires_grad)rzr{rr}rr0rns       zWav2Vec2FeatureEncoder.__init__cCs|D] }d|_qd|_dS)NF) parameters requires_gradrrzparamr/r/r0_freeze_parameterss z)Wav2Vec2FeatureEncoder._freeze_parameterscCsj|dddf}|jr"|jr"d|_|jD]<}|jr\|jr\|jr\dd}tjj|||}q(||}q(|S)NTcsfdd}|S)Ncs|Srr/inputsmoduler/r0custom_forwardszUWav2Vec2FeatureEncoder.forward..create_custom_forward..custom_forwardr/rrr/rr0create_custom_forwards z=Wav2Vec2FeatureEncoder.forward..create_custom_forward)rtrainingrrrr,r checkpoint)rz input_valuesr$Z conv_layerrr/r/r0rs   zWav2Vec2FeatureEncoder.forward)r(r)r*r+rnrrrr/r/r}r0rs rcseZdZfddZZS)Wav2Vec2FeatureExtractorcs8t|td|jjd|jjdjdtdS)Nz The class `zD` has been depreciated and will be removed in Transformers v5. Use `rz ` instead.)rmrnwarningswarnr~r( __bases__ FutureWarningrzr{r}r/r0rns  z!Wav2Vec2FeatureExtractor.__init__)r(r)r*rnrr/r/r}r0rsrcs$eZdZfddZddZZS)Wav2Vec2FeatureProjectioncsJttj|jd|jd|_t|jd|j|_ t |j |_ dS)Nr@Zeps) rmrnrrrolayer_norm_epsrLinearr projectionDropoutZfeat_proj_dropoutdropoutrr}r/r0rns z"Wav2Vec2FeatureProjection.__init__cCs&||}||}||}||fSr)rrr)rzr$Znorm_hidden_statesr/r/r0rs   z!Wav2Vec2FeatureProjection.forwardrr/r/r}r0rs rc seZdZdZdeeeeedfdd Zej eedd d Z dej e ej e e ej e ej e ej ee ej e ej e e ej fd d dZ ZS)Wav2Vec2Attentionz=Multi-headed attention from 'Attention Is All You Need' paperFT) embed_dim num_headsr is_decoderrlcst||_||_||_|||_|j||jkrNtd|jd|d|jd|_||_t j |||d|_ t j |||d|_ t j |||d|_ t j |||d|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g)rl)rmrnrrrhead_dimrHscalingrrrk_projv_projq_projout_proj)rzrrrrrlr}r/r0rns   zWav2Vec2Attention.__init__)tensorseq_lenbszcCs ||||j|jddSr)viewrrr contiguous)rzrrrr/r/r0_shapeszWav2Vec2Attention._shapeN)r$key_value_statespast_key_valuer4layer_head_maskoutput_attentionsr6cCsx|dk }|\}} } |||j} |r\|dk r\|djd|jdkr\|d} |d} n|r|||d|} |||d|} n|dk r|||d|} |||d|} tj|d| gdd} tj|d| gdd} n(|||d|} |||d|} |j r | | f}||j d|j f}|| | |j |} | j |} | j |} | d}t| | dd}|||j | |fkrtd||j | |fd||dk r ||d| |fkrtd |d| |fd|| ||j | ||}| ||j | |}tjj|dd}|dk r||j fkrhtd |j fd|| dddd| ||j | |}| ||j | |}|r| ||j | |}| ||j | |}nd}tjj||j|jd }t|| }|||j | |j fkr4td ||j | |j fd|| ||j | |j }|dd}| || |j}||}|||fS) z#Input shape: Batch x Time x ChannelNrrrr@rz$Attention weights should be of size z , but is z!Attention mask should be of size z/Head mask for a single layer should be of size )prz `attn_output` should be of size )rcrrr1rrrr,catrrrrr[ZbmmrrHr functionalsoftmaxrrrr)rzr$rrr4rrZis_cross_attentionrZtgt_lenrCZ query_statesZ key_statesZ value_statesZ proj_shapeZsrc_len attn_weightsZattn_weights_reshapedZ attn_probsZ attn_outputr/r/r0rs~       "    zWav2Vec2Attention.forward)rFT)NNNNF)r(r)r*r+r8floatrRrnr,Tensorrrrrrr/r/r}r0rs4 rcs$eZdZfddZddZZS)Wav2Vec2FeedForwardcsptt|j|_t|j|j|_ t |j t rDt |j |_n|j |_t|j|j|_t|j|_dSr)rmrnrrZactivation_dropoutintermediate_dropoutrrZintermediate_sizeintermediate_dense isinstanceZ hidden_actstrr intermediate_act_fn output_densehidden_dropoutoutput_dropoutrr}r/r0rns  zWav2Vec2FeedForward.__init__cCs6||}||}||}||}||}|Sr)rrrrrrr/r/r0rs      zWav2Vec2FeedForward.forwardrr/r/r}r0rs rcs&eZdZfddZdddZZS)Wav2Vec2EncoderLayercsftt|j|j|jdd|_t|j |_ tj |j|j d|_ t||_tj |j|j d|_dS)NFrrrrr)rmrnrrnum_attention_headsattention_dropout attentionrrrrrrrr feed_forwardfinal_layer_normrr}r/r0rns  zWav2Vec2EncoderLayer.__init__NFcCsf|}|j|||d\}}}||}||}||}|||}||}|f}|rb||f7}|SNr4r)rrrrrrzr$r4rZ attn_residualrrCoutputsr/r/r0rs     zWav2Vec2EncoderLayer.forward)NFrr/r/r}r0rs rcs8eZdZfddZdejeejedddZZ S) #Wav2Vec2EncoderLayerStableLayerNormcstt|j|j|jdd|_t|j |_ tj |j|j d|_ t||_tj |j|j d|_t|dddk r~t||_nd|_dS)NFrradapter_attn_dim)rmrnrrrrrrrrrrrrrrrgetattrWav2Vec2AttnAdapterLayer adapter_layerrr}r/r0rns   z,Wav2Vec2EncoderLayerStableLayerNorm.__init__NF)r$r4rcCsz|}||}|j|||d\}}}||}||}||||}|jdk rb|||}|f}|rv||f7}|Sr)rrrrrrrr/r/r0rs      z+Wav2Vec2EncoderLayerStableLayerNorm.forward)NF) r(r)r*rnr,rrrRrrr/r/r}r0rs rcs<eZdZfddZd ejeejeeedddZ Z S) Wav2Vec2Encodercsft|_t|_tjjjd|_ t j |_ t fddtjD|_d|_dS)Nrcsg|] }tqSr/)rrArr/r0rDsz,Wav2Vec2Encoder.__init__..Frmrnr{rpos_conv_embedrrrrrrrrrrPnum_hidden_layerslayersrrr}rr0rns   zWav2Vec2Encoder.__init__NFT)r$r4routput_hidden_states return_dictcs|rdnd}rdnd}|dk r|ddd|jd}d||<d|ddddddfj|jd}|t|jj}||jdd|jd|jd}| |} || }| |}| |}t } |j D]} |r||f}tg} |jr| |jjkrdnd } | r| r`|jrJ|jrJfd d }tjj|| ||}n| ||d }|d}| rjd }r||df}q|r||f}|stdd|||fDSt|||dS)Nr/r@rrr?rETFcsfdd}|S)Ncs|fSrr/rrrr/r0r(szNWav2Vec2Encoder.forward..create_custom_forward..custom_forwardr/rrrr0r'sz6Wav2Vec2Encoder.forward..create_custom_forwardrNNcss|]}|dk r|VqdSrr/rBvr/r/r0 Bsz*Wav2Vec2Encoder.forward..last_hidden_stater$r%) unsqueezerepeatr1torFr,finfominexpandrrrr rrKrr{ layerdroprrrtupler rzr$r4rrrZall_hidden_statesZall_self_attentionsZexpand_attention_maskZposition_embeddingsZdeepspeed_zero3_is_enabledrZdropout_probabilityZskip_the_layerrZ layer_outputsr/rr0rsd   &         zWav2Vec2Encoder.forward)NFFT) r(r)r*rnr,rrrrRrrr/r/r}r0rs rcs&eZdZfddZdddZZS) Wav2Vec2EncoderStableLayerNormcsft|_t|_tjjjd|_ t j |_ t fddtjD|_d|_dS)Nrcsg|] }tqSr/)rrArr/r0rDRsz;Wav2Vec2EncoderStableLayerNorm.__init__..Frrr}rr0rnKs  z'Wav2Vec2EncoderStableLayerNorm.__init__NFTcs|rdnd}rdnd}|dk r|ddd|jd}d||<d|ddddddfj|jd}|t|jj}||jdd|jd|jd}| |} || }| |}t } |j D]} |r||f}t g} |jr| |jjkrdnd } | r| rR|jr<|jr<fd d }tjj|| ||}n| ||d }|d}| r\d }r||df}q||}|r||f}|stdd|||fDSt|||dS)Nr/r@rrrrrETFcsfdd}|S)Ncs|fSrr/rrr/r0rsz]Wav2Vec2EncoderStableLayerNorm.forward..create_custom_forward..custom_forwardr/rrrr0rszEWav2Vec2EncoderStableLayerNorm.forward..create_custom_forwardrrcss|]}|dk r|VqdSrr/rr/r/r0rsz9Wav2Vec2EncoderStableLayerNorm.forward..r)r r r1r rFr,r r rrrr rrKrr{rrrrrrr rr/rr0rVsd   &         z&Wav2Vec2EncoderStableLayerNorm.forward)NFFTrr/r/r}r0rJs  rcs8eZdZdZfddZed ddZd ddZZS) Wav2Vec2GumbelVectorQuantizerz Vector quantization using gumbel softmax. See `[CATEGORICAL REPARAMETERIZATION WITH GUMBEL-SOFTMAX](https://arxiv.org/pdf/1611.01144.pdf) for more information. cst|j|_|j|_|j|jdkrDtd|jd|jdt t d|j|j|j|j|_ t |jd|j|j|_d|_dS)Nrz`config.codevector_dim z5 must be divisible by `config.num_codevector_groups` z for concatenationrr@r)rmrnnum_codevector_groupsrnum_codevectors_per_groupnum_varscodevector_dimrHr Parameterr,r- codevectorsrro weight_proj temperaturerr}r/r0rns z&Wav2Vec2GumbelVectorQuantizer.__init__NcCs|dk rP|ddddf|j}t||t|}|jdd|}n |jdd}ttj|t |ddd }|S)NrrgHz>r@) flattenrr1r,whereZ zeros_likerMmeanexplog)ZprobsmaskZ mask_extendedZmarginal_probs perplexityr/r/r0_compute_perplexitys (z1Wav2Vec2GumbelVectorQuantizer._compute_perplexityc Cs|j\}}}||}||||jd}|jrtjj||j dd |}t j ||||jddd}| ||}nJ|jdd} ||jd| ddd}||||jd}| ||}|||d}|d|j} | |||j|jd} | d||d} | |fS)Nr@T)tauhardrrrr)r1rrrrrrZgumbel_softmaxrrtype_asr,rr#argmaxZ new_zerosZscatter_r rrrM) rzr$rbr\r>rZcodevector_probsZcodevector_soft_distr"Zcodevector_idxZcodevectors_per_grouprr/r/r0rs:      z%Wav2Vec2GumbelVectorQuantizer.forward)N)N) r(r)r*r+rn staticmethodr#rrr/r/r}r0rs   rcs$eZdZfddZddZZS)Wav2Vec2Adaptercsptjjkr8tjj|_tj|_n d|_|_t fddt j D|_ j |_ dS)Nc3s|]}tVqdSr)Wav2Vec2AdapterLayerrArr/r0rsz+Wav2Vec2Adapter.__init__..)rmrnoutput_hidden_sizerrrprojrproj_layer_normrrPnum_adapter_layersrrrr}rr0rns    zWav2Vec2Adapter.__init__cCsr|jdk r(|jdk r(||}||}|dd}|jD]&}tj}|jrX||jkr:||}q:|dd}|Sr)r,r-rrrIrJrr)rzr$rZlayerdrop_probr/r/r0rs       zWav2Vec2Adapter.forwardrr/r/r}r0r)s r)cs$eZdZfddZddZZS)r*cs0ttj|jd|j|j|jdd|_dS)Nrr)rkr)rmrnrrrr+Zadapter_kernel_sizeadapter_stridervrr}r/r0rns zWav2Vec2AdapterLayer.__init__cCs||}tjj|dd}|S)Nrr)rvrrZglurr/r/r0rs zWav2Vec2AdapterLayer.forwardrr/r/r}r0r*s r*cs,eZdZfddZejdddZZS)rcsZt|j|_|j|_t|j|_t |j|j|_ t |_ t |j|j|_ dS)z Implements adapter modules directly with 3D tensor weight as parameters and without using ModuleList to speed up training throughput. N)rmrnr input_dimrZ hidden_dimrrnormrlinear_1ReLUact_fnlinear_2rr}r/r0rn#s  z!Wav2Vec2AttnAdapterLayer.__init__)r$cCs,||}||}||}||}|Sr)r1r2r4r5rr/r/r0r1s     z Wav2Vec2AttnAdapterLayer.forward)r(r)r*rnr,r-rrr/r/r}r0r"s rc@seZdZdZeZdZdZdZddZ de e j e feedd d Zde e j d d d ZdddZddZddZdedddZdS)Wav2Vec2PreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. wav2vec2rTc Cst|tr2|j|jd|j_d|j_nt|trp|jjj j ddd|jj j t j|jnlt|trt jj |jjddtd|jjd|jjdt j|jj dnt|trtd|jj}t jj|jj| |dt jj|jj | |dnt|t jrR|jj j d|jjd|j dk r|j j nt|t jt jfr|j j |jj d nZt|t j!rt j"|j|j dk rt|j#|j|jd}t jj|j | |ddS) zInitialize the weightsTrr)rstdrr)abNr)$rWav2Vec2ForPreTraining project_hidZreset_parameters project_qZ_is_hf_initializedrrrdataZnormal_rlZzero_rinituniform_rrrvmathsqrtrjZ in_channelsZ constant_rrZ in_featuresrr{Zinitializer_rangerrZfill_rrZkaiming_normal_r)rzrkr/r/r0 _init_weightsFs@           z%Wav2Vec2PreTrainedModel._init_weightsN)r] add_adaptercCsn|dkr|jjn|}dd}t|jj|jjD]\}}||||}q.|rjt|jjD]}||d|jj}qT|S)zH Computes the output length of the convolutional layers NcSstj|||dddS)Nfloor)Z rounding_moder)r,divr:rjrkr/r/r0_conv_out_lengthvszRWav2Vec2PreTrainedModel._get_feat_extract_output_lengths.._conv_out_lengthr)r{rEziprsrtrPr.r/)rzr]rErIrjrkrCr/r/r0 _get_feat_extract_output_lengthsmsz8Wav2Vec2PreTrainedModel._get_feat_extract_output_lengths)feature_vector_lengthr4cCs|jdddddf}|j||d}|tj}|jd}tj||f|j|jd}d|tj |jd|jd|df<| dgd dg }|S)Nr@rrEr)rFdevicer)rN) ZcumsumrKr r,longr1rQrFrNrTfliprR)rzrLr4rEZnon_padded_lengthsZoutput_lengthsr\r/r/r0"_get_feature_vector_attention_masks  "z:Wav2Vec2PreTrainedModel._get_feature_vector_attention_maskFcCst|tttfr||_dSr)rrrrr)rzrvaluer/r/r0_set_gradient_checkpointingsz3Wav2Vec2PreTrainedModel._set_gradient_checkpointingcCs|jjdkrt|jdi}|D]6\}}t|tr(|D]\}}||d||g<qBq(t|t r|j D]\}}||dd|g<qt|S)NzF has no adapter layers. Make sure to define `config.adapter_attn_dim`..lm_head) r{rrHr~Z named_modulesrrZnamed_parametersjoinWav2Vec2ForCTCrU)rzadapter_weightsrr param_namerr/r/r0 _get_adapterss   z%Wav2Vec2PreTrainedModel._get_adapterscCs<|D]}t|tr||qt|tr8||jdS)zc (Re-)initialize attention adapter layers and lm head for adapter-only fine-tuning N)modulesrrrDrWrU)rzrr/r/r0init_adapter_layerss     z+Wav2Vec2PreTrainedModel.init_adapter_layers target_langc s|jjdkrtd|d||jkr@|s@td|ddS|dd}|dd}|d d}|d d}|d d}|d d} |d d} |dd} |dtrdnd} | dk rt dt | dk rtd| } |jj } d}| dk rzt |}z&t| |||||| | |d }t|}WnTtk rB| r>Yn8tk rx| rttd| d| d|dYnX|dkrt |}z,t| |||||| | |d }tj|dd}WnHtk rYn2tk rtd| d| d|dYnX|t|t}tt|}t|dkrhtd|dd|dn*t|dkrtd|dd|d|djd}||jjkrtj|jj||j|j d|_!||j_fdd |"D}|j#|dd!||_dS)"a Load a language adapter model from a pre-trained adapter model. Parameters: target_lang (`str`): Has to be a language id of an existing adapter weight. Adapter weights are stored in the format adapter..safetensors or adapter..bin force_load (`bool`, defaults to `True`): Whether the weights shall be loaded even if `target_lang` matches `self.target_lang`. cache_dir (`Union[str, os.PathLike]`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist. resume_download (`bool`, *optional*, defaults to `False`): Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists. proxies (`Dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. local_files_only(`bool`, *optional*, defaults to `False`): Whether or not to only look at local files (i.e., do not try to download the model). token (`str` or `bool`, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, or not specified, will use the token generated when running `huggingface-cli login` (stored in `~/.huggingface`). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. To test a pull request you made on the Hub, you can pass `revision="refs/pr/". mirror (`str`, *optional*): Mirror source to accelerate downloads in China. If you are from China and have an accessibility problem, you can set this option to resolve it. Note that we do not guarantee the timeliness or safety. Please refer to the mirror site for more information. Activate the special ["offline-mode"](https://huggingface.co/transformers/installation.html#offline-mode) to use this method in a firewalled environment. Examples: ```python >>> from transformers import Wav2Vec2ForCTC, AutoProcessor >>> ckpt = "facebook/mms-1b-all" >>> processor = AutoProcessor.from_pretrained(ckpt) >>> model = Wav2Vec2ForCTC.from_pretrained(ckpt, target_lang="eng") >>> # set specific language >>> processor.tokenizer.set_target_lang("spa") >>> model.load_adapter("spa") ``` NzCannot load_adapter for - if `config.adapter_attn_dim` is not defined.z#Adapter weights are already set to rT cache_dirforce_downloadFresume_downloadproxieslocal_files_onlytokenuse_auth_tokenrevisionuse_safetensorszVThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers.zV`token` and `use_auth_token` are both specified. Please set only the argument `token`.)filenamerarbrcrdrergr`zCan't load the model for 'z'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'z=' is the correct path to a directory containing a file named cpu)Z map_locationrzThe adapter weights z has unexpected keys: z, z has missing keys: zlm_head.weightrNrFcs i|]\}}|||qSr/)r )rBrCrrXr/r0 lsz8Wav2Vec2PreTrainedModel.load_adapter..)strict)$r{rrHr^loggerwarningpoprrrrZ _name_or_pathWAV2VEC2_ADAPTER_SAFE_FILEformatrsafe_load_fileEnvironmentError ExceptionWAV2VEC2_ADAPTER_PT_FILEr,loadrZsetkeysrUrVr1 vocab_sizerrr+rNrFrUitemsZload_state_dict)rzr^ force_loadkwargsr`rarbrcrdrerfrgrhZmodel_path_or_idZ state_dictfilepathZ weight_pathZunexpected_keysZ missing_keysZtarget_vocab_sizer/rlr0 load_adapters?                 z$Wav2Vec2PreTrainedModel.load_adapter)N)N)F)T)r(r)r*r+r config_classZbase_model_prefixZmain_input_nameZsupports_gradient_checkpointingrDrr, LongTensorr8rrRrKrQrSrZr\rrr/r/r/r0r6;s((     r6a Wav2Vec2 was proposed in [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://arxiv.org/abs/2006.11477) by Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, Michael Auli. This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving etc.). This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`Wav2Vec2Config`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a `.flac` or `.wav` audio file into an array of type `List[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoProcessor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`Wav2Vec2Processor.__call__`] for details. attention_mask (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing convolution and attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) `attention_mask` should only be passed if the corresponding processor has `config.return_attention_mask == True`. For all models whose processor has `config.return_attention_mask == False`, such as [wav2vec2-base](https://huggingface.co/facebook/wav2vec2-base-960h), `attention_mask` should **not** be passed to avoid degraded performance when doing batched inference. For such models `input_values` should simply be padded with 0 and passed without `attention_mask`. Be aware that these models also yield slightly different results depending on whether `input_values` is padded or not. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zbThe bare Wav2Vec2 Model transformer outputting raw hidden-states without any specific head on top.c seZdZedfdd ZddZddZdeje eje ej d d d Z e e eeeed ed de eje eje eje ee ee eeeefdddZZS) Wav2Vec2Modelrcst|||_t||_t||_|jdks:|jdkrRt t |j |_|jrdt||_n t||_|jr|t|nd|_|dS)Nr)rmrnr{rfeature_extractorrfeature_projectionmask_time_probmask_feature_probrrr,r-rr@masked_spec_embedZdo_stable_layer_normrencoderrrEr)adapter post_initrr}r/r0rns     zWav2Vec2Model.__init__cCstdt|dSz Calling this function will disable the gradient computation for the feature encoder so that its parameters will not be updated during training. The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5.Please use the equivalent `freeze_feature_encoder` method instead.Nrrrfreeze_feature_encoderrzr/r/r0freeze_feature_extractors z&Wav2Vec2Model.freeze_feature_extractorcCs|jdS Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training. N)rrrr/r/r0rsz$Wav2Vec2Model.freeze_feature_encoderN)r$rbr4cCst|jdds|S|\}}}|dk r<|j|j||<nZ|jjdkr|jrt||f|jj|jj ||jj d}t j ||j t jd}|j|j||<|jjdkr|jrt||f|jj|jj|jjd}t j ||j t jd}|dddfd|d}d||<|S) z Masks extracted features along time axis and/or along feature axis according to [SpecAugment](https://arxiv.org/abs/1904.08779). Zapply_spec_augmentTNr)r2r3r4r5rk)r2r3r5r@)rr{rcrr rFrrr_Zmask_time_lengthZmask_time_min_masksr,rrNrRrZmask_feature_lengthZmask_feature_min_masksr)rzr$rbr4r\r>rZmask_feature_indicesr/r/r0_mask_hidden_statess4 z!Wav2Vec2Model._mask_hidden_statesaudior output_typermodalityexpected_output)rr4rbrrrr6c Cs|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}||}|dd}|dk rp|j|jd|dd}||\}}|j |||d}|j |||||d} | d}|j dk r| |}|s||f| ddSt ||| j | jdS) NrrFrM)rbr4r4rrrr)rextract_featuresr$r%)r{rruse_return_dictrrrQr1rrrrrr$r%) rzrr4rbrrrrr$Zencoder_outputsr/r/r0rsH    zWav2Vec2Model.forward)NN)NNNNN)r(r)r*rrnrrr,r-rrrrWAV_2_VEC_2_INPUTS_DOCSTRINGr_CHECKPOINT_FOR_DOCr_CONFIG_FOR_DOC_EXPECTED_OUTPUT_SHAPErrRrrrrr/r/r}r0rs@   .  rz5Wav2Vec2 Model with a quantizer and `VQ` head on top.c seZdZedfdd ZedddZddZd d Ze de j e j e j ed d dZ e eeeeddee jee jee jee jeeeeeeeeefdddZZS)r;rcs^t|t||_t|j|_t||_ t |j |j |_ t |j|j |_|dSr)rmrnrr7rrZfeat_quantizer_dropoutdropout_featuresr quantizerrrZproj_codevector_dimr<rr=rrr}r/r0rnEs   zWav2Vec2ForPreTraining.__init__)rcCs ||j_dS)zb Set the Gumbel softmax temperature to a given value. Only necessary for training N)rr)rzrr/r/r0set_gumbel_temperatureRsz-Wav2Vec2ForPreTraining.set_gumbel_temperaturecCstdt|dSrrrr/r/r0rXs z/Wav2Vec2ForPreTraining.freeze_feature_extractorcCs|jjdSrr7rrrr/r/r0rdsz-Wav2Vec2ForPreTraining.freeze_feature_encoder皙?)target_featuresnegative_featurespredicted_featuresrcCs<tj||gdd}tj||dd|}||}|S)z Compute logits for contrastive loss based using cosine similarity as the distance measure between `[positive_feature, negative_features]` and `[predicted_features]`. Additionally, temperature can be applied. rrr@)r,rZcosine_similarityrr&)rrrrlogitsr/r/r0compute_contrastive_logitsks z1Wav2Vec2ForPreTraining.compute_contrastive_logits)rrN)rr4rbrerrrr6c Cs|dk r |n|jj}|dk r(|tj}|j||||||d}||d} ||d} |dk rx|j| j d|dd}|j | |d\} } | | } d} }}|dk r| j \}}}| d|| d}| ||d|d ddd }|| dddf|| |jj}| |kd}|r8td |dd|<|dd d|d}d| d dd}tjj||d d}|jj|jj}|| ||}||jj|} |s| dk r| | | | f|d dS| | | f|d dSt| | | | |j |j!||dS)a mask_time_indices (`torch.BoolTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices to mask extracted features for contrastive loss. When in training mode, model learns to predict masked extracted features in *config.proj_codevector_dim* space. sampled_negative_indices (`torch.BoolTensor` of shape `(batch_size, sequence_length, num_negatives)`, *optional*): Indices indicating which quantized target vectors are used as negative sampled vectors in contrastive loss. Required input for pre-training. Returns: Example: ```python >>> import torch >>> from transformers import AutoFeatureExtractor, Wav2Vec2ForPreTraining >>> from transformers.models.wav2vec2.modeling_wav2vec2 import _compute_mask_indices, _sample_negative_indices >>> from datasets import load_dataset >>> feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base") >>> model = Wav2Vec2ForPreTraining.from_pretrained("facebook/wav2vec2-base") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> input_values = feature_extractor(ds[0]["audio"]["array"], return_tensors="pt").input_values # Batch size 1 >>> # compute masked indices >>> batch_size, raw_sequence_length = input_values.shape >>> sequence_length = model._get_feat_extract_output_lengths(raw_sequence_length).item() >>> mask_time_indices = _compute_mask_indices( ... shape=(batch_size, sequence_length), mask_prob=0.2, mask_length=2 ... ) >>> sampled_negative_indices = _sample_negative_indices( ... features_shape=(batch_size, sequence_length), ... num_negatives=model.config.num_negatives, ... mask_time_indices=mask_time_indices, ... ) >>> mask_time_indices = torch.tensor(data=mask_time_indices, device=input_values.device, dtype=torch.long) >>> sampled_negative_indices = torch.tensor( ... data=sampled_negative_indices, device=input_values.device, dtype=torch.long ... ) >>> with torch.no_grad(): ... outputs = model(input_values, mask_time_indices=mask_time_indices) >>> # compute cosine similarity between predicted (=projected_states) and target (=projected_quantized_states) >>> cosine_sim = torch.cosine_similarity(outputs.projected_states, outputs.projected_quantized_states, dim=-1) >>> # show that cosine similarity is much higher than random >>> cosine_sim[mask_time_indices.to(torch.bool)].mean() > 0.5 tensor(True) >>> # for contrastive loss training model should be put into train mode >>> model = model.train() >>> loss = model( ... input_values, mask_time_indices=mask_time_indices, sampled_negative_indices=sampled_negative_indices ... ).loss ```N)r4rrrbrrrFrM)rbr@rrz-infirM) reduction)r r!r"r#r$r%r&r')"r{rr r,rRr7r<rrQr1rr=rrOZpermuterZcontrastive_logits_temperatureallanyrrr[rcrrrZ cross_entropyrrrMZdiversity_loss_weightrr$r%)rzrr4rbrerrrrZtransformer_featuresrZquantized_featuresr#r r&r'r\r>rZnegative_quantized_featuresrZ neg_is_postargetZnum_codevectorsr/r/r0rsE            zWav2Vec2ForPreTraining.forward)r)NNNNNN)r(r)r*rrnr8rrrr(r,r-rrrrrrrrZ BoolTensorrRrrrrr/r/r}r0r;Cs<    r;z6Wav2Vec2 Model with a `language modeling` head on top.c sbeZdZfddZeedejeej ee ee ee eej e e efdddZZS)Wav2Vec2ForMaskedLMcsNt|tdtt||_t|j |_ t |j |j |_|dS)NzSThe class `Wav2Vec2ForMaskedLM` is deprecated. Please use `Wav2Vec2ForCTC` instead.)rmrnrrrrr7rr final_dropoutrrrr{rUrrr}r/r0rn#s  zWav2Vec2ForMaskedLM.__init__Nrr4rrrlabelsr6c Csn|dk r |n|jj}|j||||d}|d}||}||} |s\| f|dd} | St| |j|jdS)N)rrrrr)rr$r%)r{rr7rrUr r$r%) rzrr4rrrrrr$routputr/r/r0r1s   zWav2Vec2ForMaskedLM.forward)NNNNN)r(r)r*rnrrr,r-rrrRrrrr rrr/r/r}r0r!s   rzfWav2Vec2 Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC).c seZdZdeedfdd ZddZddZd d Zd d Z e e e e eeeed deejeejeeeeeeeejeeefdddZZS)rWNr]cs~t|t||_t|j|_||_|j dkrFt d|j dt |dr\|j r\|jn|j}t||j |_|dS)NzYou are trying to instantiate z with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `Wav2Vec2ForCTC.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration.rE)rmrnrr7rrrrr^r{rHr~rrEr+rrrUr)rzr{r^r+r}r/r0rnTs    zWav2Vec2ForCTC.__init__cCsr|j}|dk r2t|jdddkr2td|dn<|dkrXt|jdddk rXtdn|dk rn|j|dddS)a' This method overwrites [`~PreTrainedModel.tie_weights`] so that adapter weights can be correctly loaded when passing `target_lang=...` to `from_pretrained(...)`. This method is **not** supposed to be called by the user and is prone to be changed in the future. NrzCannot pass `target_lang`: r_z)By default `target_lang` is set to 'eng'.T)r})r^rr{rHroinfor)rzr^r/r/r0 tie_weightsks  zWav2Vec2ForCTC.tie_weightscCstdt|dSrrNrrr/r/r0rs z'Wav2Vec2ForCTC.freeze_feature_extractorcCs|jjdSrrrr/r/r0rsz%Wav2Vec2ForCTC.freeze_feature_encodercCs|jD] }d|_q dSz Calling this function will disable the gradient computation for the base model so that its parameters will not be updated during training. Only the classification head will be updated. FNr7rrrr/r/r0freeze_base_modelsz Wav2Vec2ForCTC.freeze_base_model)rrrr expected_lossrc Csf|dk r |n|jj}|j|||||d}|d}||}||} d} |dk r"||jjkrttd|jj|dk r|ntj |tj d}| | d tj } |dk} | d} || }tjj| dtjddd}tjjjd d ,tjj||| | |jj|jj|jjd } W5QRX|sR| f|td}| dk rN| f|S|St| | |j|jd S) a labels (`torch.LongTensor` of shape `(batch_size, target_length)`, *optional*): Labels for connectionist temporal classification. Note that `target_length` has to be smaller or equal to the sequence length of the output logits. Indices are selected in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size - 1]`. Nrrz$Label values must be <= vocab_size: rEr@)rrFrF)enabled)blankrZ zero_infinityr rr$r%)r{rr7rrUr9r{rHr,Z ones_likerOrKrMr Z masked_selectrrZ log_softmaxZfloat32rbackendsZcudnnflagsZctc_lossZ pad_token_idZctc_loss_reductionZctc_zero_infinity_HIDDEN_STATES_START_POSITIONr r$r%)rzrr4rrrrrr$rr r]Z labels_maskZtarget_lengthsZflattened_targetsZ log_probsrr/r/r0rsR      zWav2Vec2ForCTC.forward)N)NNNNN)r(r)r*rrrnrrrrrrrrr r_CTC_EXPECTED_OUTPUT_CTC_EXPECTED_LOSSr,rrRrrrrr/r/r}r0rWOs6   rWz Wav2Vec2 Model with a sequence classification head on top (a linear layer over the pooled output) for tasks like SUPERB Keyword Spotting. c seZdZfddZddZddZddZeee e e e d e ed deejeejeeeeeeeejeee fd d dZZS)!Wav2Vec2ForSequenceClassificationcst|t|dr$|jr$tdt||_|jd}|jrTt t |||_ t |j|j|_t |j|j|_|dS)NrEz_Sequence classification does not support the use of Wav2Vec2 adapters (config.add_adapter=True)r)rmrnrrErHrr7ruse_weighted_layer_sumrrr,rV layer_weightsrrZclassifier_proj_size projector num_labels classifierrrzr{ num_layersr}r/r0rns   z*Wav2Vec2ForSequenceClassification.__init__cCstdt|dSrrrr/r/r0rs z:Wav2Vec2ForSequenceClassification.freeze_feature_extractorcCs|jjdSrrrr/r/r0rsz8Wav2Vec2ForSequenceClassification.freeze_feature_encodercCs|jD] }d|_q dSrrrr/r/r0rsz3Wav2Vec2ForSequenceClassification.freeze_base_modelr)rrrrrrNrcCsf|dk r |n|jj}|jjr dn|}|j|||||d}|jjr|t}tj|dd}tjj |j dd} || dddj dd}n|d}| |}|dkr|jdd} n<||jd|} d|| <|j dd| j dd dd} || } d} |dk r"t}|| d|jj| d} |sR| f|td}| dk rN| f|S|St| | |j|jd S)  labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NTrrrr@rrr)r{rrr7rr,stackrrrrrrMrrrQr1rrrrr$r%)rzrr4rrrrrr$ norm_weightsZ pooled_outputZ padding_maskrr loss_fctrr/r/r0rsF     z)Wav2Vec2ForSequenceClassification.forward)NNNNN)r(r)r*rnrrrrrr_SEQ_CLASS_CHECKPOINTrr_SEQ_CLASS_EXPECTED_OUTPUT_SEQ_CLASS_EXPECTED_LOSSrr,rrRrrrrr/r/r}r0rs6    rzd Wav2Vec2 Model with a frame classification head on top for tasks like Speaker Diarization. c seZdZfddZddZddZddZeee e e e d e d deejeejeejeeeeeeeee fd d dZZS)#Wav2Vec2ForAudioFrameClassificationcszt|t|dr$|jr$tdt||_|jd}|jrTt t |||_ t |j|j|_|j|_|dS)NrEzbAudio frame classification does not support the use of Wav2Vec2 adapters (config.add_adapter=True)r)rmrnrrErHrr7rrrrr,rVrrrrr init_weightsrr}r/r0rnms   z,Wav2Vec2ForAudioFrameClassification.__init__cCstdt|dSrrrr/r/r0r}s z@皙?csFtt|||_||_||_tjt ||dd|_ t |_ dS)NT)r) rmrrnscalemarginrrrr,Zrandnrrr )rzr0rrrr}r/r0rns zAMSoftmaxLoss.__init__c Csx|}tjj|jdd}tjj|dd}t||}||j}tj||j }|j t | ||}| ||}|S)Nrrr)rrr normalizerr,mmrZone_hotrrrrRr ) rzr$rrZ cos_thetapsiZonehotrr r/r/r0rs   zAMSoftmaxLoss.forward)rrrr/r/r}r0rsrcs&eZdZdfdd ZddZZS) TDNNLayerrcsvt|dkr |j|dn|j||_|j||_|j||_|j||_t |j|j|j|_ t |_ dS)Nrr)rmrntdnn_dimrprq tdnn_kernelrjZ tdnn_dilationdilationrrkernelr3rxryr}r/r0rns "   zTDNNLayer.__init__cCsV|d}tjj||j|jfd|jf|jdfd}|dd}||}| |}|S)Nr)rkrr) r rrZunfoldrjrprrrrxrr/r/r0rs     zTDNNLayer.forward)rrr/r/r}r0rs rzl Wav2Vec2 Model with an XVector feature extraction head on top for tasks like Speaker Verification. c seZdZfddZddZddZddZeej e fd d d Z e e eeeed ed deejeejeeeeeeeejeeefdddZZS)Wav2Vec2ForXVectorcstt|_jd}jr.r@r)rmrnrr7rrrrr,rVrrrrrrPrUrtdnnZxvector_output_dimrrrr objectiver)rzr{rZ tdnn_layersr}rr0rn s    zWav2Vec2ForXVector.__init__cCstdt|dSrrrr/r/r0r! s z+Wav2Vec2ForXVector.freeze_feature_extractorcCs|jjdSrrrr/r/r0r- sz)Wav2Vec2ForXVector.freeze_feature_encodercCs|jD] }d|_q dSrrrr/r/r0r4 sz$Wav2Vec2ForXVector.freeze_base_model)r]cCs&dd}|jjD]}|||d}q|S)z? Computes the output length of the TDNN layers cSs|||dS)Nrr/rHr/r/r0rIA szEWav2Vec2ForXVector._get_tdnn_output_lengths.._conv_out_lengthr)r{r)rzr]rIrjr/r/r0_get_tdnn_output_lengths< s z+Wav2Vec2ForXVector._get_tdnn_output_lengthsrrNrcCs|dk r |n|jj}|jjr dn|}|j|||||d}|jjr|t}tj|dd}tjj |j dd} || dddj dd}n|d}| |}|jD] } | |}q|dkr|jdd} |jdd} n||j dd} || }g} g} t|D]D\}}| ||d|fjdd| ||d|fjddqt| } t| } tj| | gdd}||}||}d}|dk r|||}|s||f|td}|dk r|f|S|St||||j|jdS) rNTrrrr@r)r rZ embeddingsr$r%)r{rrr7rr,rrrrrrrMrrrr8rKr enumeraterXrrrrrr$r%)rzrr4rrrrrr$rZ tdnn_layerZ mean_featuresZ std_featuresZfeat_extract_output_lengthsZtdnn_output_lengthsrlengthZstatistic_poolingZoutput_embeddingsrr rr/r/r0rK s\            zWav2Vec2ForXVector.forward)NNNNN)r(r)r*rnrrrrr,rr8rrrr_XVECTOR_CHECKPOINTrr_XVECTOR_EXPECTED_OUTPUTrrrRrrrr/r/r}r0r s6    r)Nr)N)er+rAr dataclassesrtypingrrrnumpyrIr,Ztorch.utils.checkpointrZtorch.nnrZ activationsr Zintegrations.deepspeedr Zmodeling_outputsr r r rrrrZmodeling_utilsrrrrrrrrrrZconfiguration_wav2vec2rrwrrZsafetensors.torchrrtZ get_loggerr(rorrrrrrrrrrrrrZ)WAV_2_VEC_2_PRETRAINED_MODEL_ARCHIVE_LISTrr8rrZndarrayr_rgModulerhrrrrrrrrrrrrrrr)r*rr6ZWAV_2_VEC_2_START_DOCSTRINGrrr;rrWrrrrrr/r/r/r0s     $ (     -  x $' 5 "-W[L:&^-tj