U 9%eǂ@sdZddlZddlmZddlmZmZmZmZddl Z ddl Z ddl m Z ddl m Z ddlmZmZmZmZmZd d lmZeeZd Zd d gZeGdddeZeGdddeZeGdddeZGddde jZGddde jZ Gddde jZ!Gddde jZ"Gddde jZ#Gddde jZ$Gd d!d!e jZ%Gd"d#d#e jZ&Gd$d%d%e jZ'Gd&d'd'e Z(d(Z)d)Z*ed*e)Gd+d,d,e(Z+dS)-z PyTorch EnCodec model.N) dataclass)ListOptionalTupleUnion)nn)PreTrainedModel) ModelOutputadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) EncodecConfigrzfacebook/encodec_24khzzfacebook/encodec_48khzc@s.eZdZUdZdZejed<dZejed<dS) EncodecOutputai Args: audio_codes (`torch.FloatTensor` of shape `(batch_size, nb_chunks, chunk_length)`, *optional*): Discret code embeddings computed using `model.encode`. audio_values (`torch.FlaotTensor` of shape `(batch_size, sequence_length)`, *optional*) Decoded audio values, obtained using the decoder part of Encodec. N audio_codes audio_values) __name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rrrk/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/transformers/models/encodec/modeling_encodec.pyr2s rc@s.eZdZUdZdZejed<dZejed<dS)EncodecEncoderOutputa Args: audio_codes (`torch.FloatTensor` of shape `(batch_size, nb_chunks, chunk_length)`, *optional*): Discret code embeddings computed using `model.encode`. audio_scales (`torch.Tensor` of shape `(batch_size, nb_chunks)`, *optional*): Scaling factor for each `audio_codes` input. This is used to unscale each chunk of audio when decoding. Nr audio_scales) rrrrrrrrrrrrrr@s rc@s eZdZUdZdZejed<dS)EncodecDecoderOutputz Args: audio_values (`torch.FloatTensor` of shape `(batch_size, segment_length)`, *optional*): Decoded audio values, obtained using the decoder part of Encodec. Nr)rrrrrrrrrrrrrNs rcszeZdZdZdeeeeedfdd Zedejeeeeddd Z edeje eefe e d d dZ ddZZS) EncodecConv1dz;Conv1d with asymmetric or causal padding and normalization.r) in_channels out_channels kernel_sizestridedilationc st|j|_|j|_|j|_|jdkr 1 and dilation > 1 (kernel_size=z stride=z , dilation=z).r%r'r()super__init__use_causal_convcausalpad_mode norm_type ValueErrorloggerwarningrConv1dconvutilsr' GroupNormnorm)selfconfigr!r"r#r$r% __class__rrr,\s"     zEncodecConv1d.__init__r) hidden_statesr#r$ padding_totalreturncCs@|jd}||||d}t|d|||}||S)zSee `pad_for_conv1d`.r)shapemathceil)r=r#r$r>lengthZn_framesZ ideal_lengthrrr_get_extra_padding_for_conv1dvs z+EncodecConv1d._get_extra_padding_for_conv1dzero)r=paddingsmodevaluec Cs|jd}|\}}|dks,tj||||St||}d}||kr`||d}tj|d|f}tj||||} | jd|} | dd| fS)zTiny wrapper around torch.nn.functional.pad, just to allow for reflect padding on small input. If this is the case, we insert extra 0 padding to the right before the reflection happens. r@Zreflectrr.N)rAr functionalpadmax) r=rHrIrJrD padding_left padding_rightZmax_padZ extra_padpaddedendrrr_pad1ds   zEncodecConv1d._pad1dc Cs|jjd}|jjd}|jjd}|d|d}||}|||||}|jrj|j|||f|jd}n*|d}||}|j||||f|jd}||}|jdkr| |}|S)Nrr)rIr() r5r#r$r%rEr.rRr/r0r8) r9r=r#r$r%r>Z extra_paddingrOrNrrrforwards&       zEncodecConv1d.forward)rr)r)rFrG)rrrrintr, staticmethodrTensorrErstrfloatrRrT __classcell__rrr;rr Ys, "r cs6eZdZdZdeeeedfdd ZddZZS) EncodecConvTranspose1dzDConvTranspose1d with asymmetric or causal padding and normalization.r)r!r"r#r$cst|j|_|j|_|j|_|jdkr||}|jrVt||j}n|d}||}|j d|}|d||f}|S)Nrr(rSr@.) r5r#r$r0r8r.rBrCr]rA)r9r=r#r$r>rOrNrQrrrrTs     zEncodecConvTranspose1d.forward)r)rrrrrUr,rTrZrrr;rr[sr[cs(eZdZdZfddZddZZS) EncodecLSTMzz LSTM without worrying about the hidden state, nor the layout of the data. Expects input as convolutional layout. cs tt|||j|_dSN)r+r,rLSTMZnum_lstm_layerslstm)r9r: dimensionr;rrr,s zEncodecLSTM.__init__cCs2|ddd}||d|}|ddd}|S)NrSrr)permutera)r9r=rrrrTszEncodecLSTM.forward)rrrrr,rTrZrrr;rr^s r^cs6eZdZdZeeeedfdd ZddZZ S)EncodecResnetBlockz> Residual block from SEANet model as used by EnCodec. )r:dim dilationsc st|jdf}t|t|kr,td||j}g}tt||D]Z\}\}} |dkr`|n|} |t|dkrx|n|} |t g7}|t || | || dg7}qHt ||_ |j rt |||dd|_n t|_dS)Nrz7Number of kernel sizes should match number of dilationsrr*)r#)r+r,Zresidual_kernel_sizelenr1compress enumerateziprELUr ModuleListblockZuse_conv_shortcutshortcutZIdentity) r9r:rerfZ kernel_sizesZhiddenrmir#r%Zin_chsZout_chsr;rrr,s    zEncodecResnetBlock.__init__cCs&|}|jD] }||}q |||Sr_)rmrn)r9r=residuallayerrrrrT s  zEncodecResnetBlock.forward) rrrrrrUrr,rTrZrrr;rrdsrdcs.eZdZdZedfdd ZddZZS)EncodecEncoderz"SEANet encoder as used by EnCodec.r:c stt||j|j|jg}d}t|jD]p}||j}t|j D] }|t |||j |dgg7}qF|t g7}|t|||d|d|dg7}|d9}q.|t|||jg7}|t g7}|t|||j|j|jg7}t ||_dS)NrrSr#r$)r+r,r audio_channels num_filtersr#reversedupsampling_ratiosrangenum_residual_layersrddilation_growth_raterrkr^ hidden_sizelast_kernel_sizerllayers)r9r:modelscalingratio current_scalejr;rrr,s    zEncodecEncoder.__init__cCs|jD] }||}q|Sr_r~r9r=rqrrrrT,s  zEncodecEncoder.forwardrrrrrr,rTrZrrr;rrrsrrcs.eZdZdZedfdd ZddZZS)EncodecDecoderz"SEANet decoder as used by EnCodec.rsc sttdt|j}t||j||j|jg}|t |||jg7}|jD]t}||j}|t g7}|t |||d|d|dg7}t |jD]$}|t||d|j|dfg7}q|d}qR|t g7}|t||j|j|jg7}t ||_dS)NrSrtr)r+r,rUrgrxr r|rvr#r^rrkr[ryrzrdr{rur}rlr~)r9r:rrrrrr;rrr,5s    " zEncodecDecoder.__init__cCs|jD] }||}q|Sr_rrrrrrTNs  zEncodecDecoder.forwardrrrr;rr2srcs>eZdZdZedfdd ZddZddZd d ZZ S) EncodecEuclideanCodebookz!Codebook with Euclidean distance.rscsjtt|j|j}|j|_|dtdg|dt|j|d||d|dS)NinitedTZ cluster_sizeembedZ embed_avg) r+r,rzeros codebook_sizeZ codebook_dimZregister_bufferrWclone)r9r:rr;rrr,Ws  z!EncodecEuclideanCodebook.__init__cCsV|j}|djddd}|d|||djddd }|jddj}|S)NrSrTkeepdimrr@)re)rtpowsumrMindices)r9r=rZ scaled_statesdist embed_indrrrquantizebs  &z!EncodecEuclideanCodebook.quantizecCs8|j}|d|df}||}|j|dd}|S)Nr@)rAZreshaperview)r9r=rArrrrencodeis  zEncodecEuclideanCodebook.encodecCstj||j}|Sr_)rrKZ embeddingrr9rrrrrdecodesszEncodecEuclideanCodebook.decode) rrrrrr,rrrrZrrr;rrTs   rcs6eZdZdZedfdd ZddZddZZS) EncodecVectorQuantizationzY Vector quantization implementation. Currently supports only euclidean distance. rscstt||_dSr_)r+r,rcodebookr9r:r;rrr,}s z"EncodecVectorQuantization.__init__cCs|ddd}|j|}|SNrrSr)rcrr)r9r=Zembed_inrrrrs z EncodecVectorQuantization.encodecCs|j|}|ddd}|Sr)rrrcrrrrrs z EncodecVectorQuantization.decode) rrrrrr,rrrZrrr;rrxsrcsleZdZdZedfdd ZdeeedddZ de j eee j d d d Z e j e j d d dZ ZS)EncodecResidualVectorQuantizerzResidual Vector Quantizer.rscsFtj|_j|_j|_tfddtjD|_dS)Ncsg|] }tqSr)r).0_rsrr sz;EncodecResidualVectorQuantizer.__init__..) r+r,r frame_ratenum_quantizersrrlryr~rr;rsrr,s  z'EncodecResidualVectorQuantizer.__init__N) bandwidthr?cCsHt|j|j}|j}|dk rD|dkrDttdt|d|}|S)z:Return num_quantizers based on specified target bandwidth.NrGri)rBlog2rrrrUrMfloor)r9rZbw_per_qrrrr get_num_quantizers_for_bandwidths z?EncodecResidualVectorQuantizer.get_num_quantizers_for_bandwidth) embeddingsrr?c CsZ||}|}g}|jd|D]*}||}||}||}||q t|} | S)z Encode a given input tensor with the specified frame rate at the given bandwidth. The RVQ encode method sets the appropriate number of quantizers to use and returns indices for each quantizer. N)rr~rrappendrstack) r9rrrrpZ all_indicesrqr quantizedZ out_indicesrrrrs     z%EncodecResidualVectorQuantizer.encode)codesr?cCsBtjd|jd}t|D]$\}}|j|}||}||}q|S)z7Decode the given codes to the quantized representation.rG)device)rZtensorrrir~r)r9rZ quantized_outrorrqrrrrrs    z%EncodecResidualVectorQuantizer.decode)N)N)rrrrrr,rrYrUrrrWrrrZrrr;rrs rc@s2eZdZdZeZdZdZdZddZ d dd Z d S) EncodecPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. Zencodec input_valuesTcCs\t|tjr<|jjjd|jjd|jdk r8|jj nt|tj tj frj|jj |jj dnt|tj rtj|j|jdk rt|j|j|jd}tjj|j| |dnt|tjr|jjjd|jjd|jdk rX|jj|j nRt|tjrX|D]:\}}d|kr `padding_mask` should always be passed, unless the input was truncated or not padded. This is because in order to process tensors effectively, the input audio should be padded so that `input_length % stride = step` with `step = chunk_length-stride`. This ensures that all chunks are of the same shape bandwidth (`float`, *optional*): The target bandwidth. Must be one of `config.target_bandwidths`. If `None`, uses the smallest possible bandwidth. bandwidth is represented as a thousandth of what it is, e.g. 6kbps bandwidth is represented as `bandwidth == 6.0` audio_codes (`torch.FloatTensor` of shape `(batch_size, nb_chunks, chunk_length)`, *optional*): Discret code embeddings computed using `model.encode`. audio_scales (`torch.Tensor` of shape `(batch_size, nb_chunks)`, *optional*): Scaling factor for each `audio_codes` input. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z%The EnCodec neural audio codec model.c sfeZdZedfdd ZddZddZeje e e eje ejfdd d Z dejeje e e eee eje ejfefd d dZeeeje dddZdeje ejejdddZdejeje eje eee ejejfefdddZeeeeeddeje eje e e eje eje eee ejejfefdddZZS) EncodecModelrscsjt|||_t||_t||_t||_t t |jj |_ d|j |jj kr^td|dS)NrSz'The codebook_size must be a power of 2.)r+r,r:rrencoderrdecoderr quantizerrUrBrrZbits_per_codebookr1Z post_initrr;rrr,s    zEncodecModel.__init__cCs|jSr_)rr9rrr get_encoder(szEncodecModel.get_encodercCs|jSr_)rrrrr get_decoder+szEncodecModel.get_decoder)rr padding_maskr?c Cs|jd}||jj}|jjdk rJ|d|jjkrJtd|d|jjd}|jjr||}tj|ddd|jd}|d j ddd  d }||}| |}|j ||} | d d} | |fS) z Encodes the given input using the underlying VQVAE. If `config.normalize` is set to `True` the input is first normalized. The padding mask is required to compute the correct scale. r@Ngh㈵>zDuration of frame (z) is longer than chunk rTrrS)rerg:0yE>r)rAr:Z sampling_rateZchunk_length_s RuntimeError normalizerrrrrrrr transpose) r9rrrrDdurationscalemonorrrrr _encode_frame.s    zEncodecModel._encode_frameN)rrr return_dictr?cCsl|dk r |n|jj}|dkr(|jjd}||jjkrNtd|d|jjd|j\}}}|dksj|dkrxtd||jj}|dkr|}|} n|jj} |dkrt| }g} g} || } || | dkrtd t d|| | D]`} |d | | |f }|dddd| | |f}| |||\}}| || |qt | } |sb| | fSt| | S) a Encodes the input audio waveform into discrete codes. Args: input_values (`torch.Tensor` of shape `(batch_size, channels, sequence_length)`): Float values of the input audio waveform. padding_mask (`torch.Tensor` of shape `(batch_size, channels, sequence_length)`): Padding mask used to pad the `input_values`. bandwidth (`float`, *optional*): The target bandwidth. Must be one of `config.target_bandwidths`. If `None`, uses the smallest possible bandwidth. bandwidth is represented as a thousandth of what it is, e.g. 6kbps bandwidth is represented as bandwidth == 6.0 Returns: A list of frames containing the discrete encoded codes for the input audio waveform, along with rescaling factors for each chunk when `normalize` is True. Each frames is a tuple `(codebook, scale)`, with `codebook` of shape `[batch_size, num_codebooks, frames]`. Nrz)This model doesn't support the bandwidth z. Select one of .rrSz1Number of audio channels must be 1 or 2, but got zkThe input length is not properly padded for batched chunked decoding. Make sure to pad the input correctly..)r:rZtarget_bandwidthsr1rA chunk_length chunk_strider ones_likeboolryrrrr)r9rrrrrZchannelsZ input_lengthrr$Zencoded_framesscalesstepoffsetmaskframeZ encoded_framerrrrrHsD      zEncodecModel.encode)framesr$c CsLt|dkrtd|dj}|dj}|djdd}|t|d|djd}|djd}tjdd|d||ddd}d|d}tj|||d} tj||f||d} d} |D]^} | jd}| d| | |f|d|| 7<| | | ||d|7<| |7} q| dkrDtd | d | | S) Nrz!`frames` cannot be an empty list.r@rrS)rdtypeg?.z7`sum_weight` minimum element must be bigger than zero: `) rgr1rrrArZlinspaceabsrmin) rr$rrrA total_sizeZ frame_lengthZtime_vecrZ sum_weightoutrrrrr_linear_overlap_adds(     (  z EncodecModel._linear_overlap_add)rrr?cCs@|dd}|j|}||}|dk r<||ddd}|S)Nrrr@)rrrrr)r9rrroutputsrrr _decode_frames    zEncodecModel._decode_frame)rrrrr?c Cs|p |jj}|jj}|dkrPt|dkr:tdt|||d|d}nBg}t||D]\}} ||| } || q^|||jj pd}|dk r|j d|j dkr|dd|j df}|s|fSt |S)aN Decodes the given frames into an output audio waveform. Note that the output might be a bit bigger than the input. In that case, any extra steps at the end can be trimmed. Args: audio_codes (`torch.FloatTensor` of shape `(batch_size, nb_chunks, chunk_length)`, *optional*): Discret code embeddings computed using `model.encode`. audio_scales (`torch.Tensor` of shape `(batch_size, nb_chunks)`, *optional*): Scaling factor for each `audio_codes` input. padding_mask (`torch.Tensor` of shape `(batch_size, channels, sequence_length)`): Padding mask used to pad the `input_values`. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NrzExpected one frame, got rr@.) r:rrrgr1rrjrrrrAr) r9rrrrrrZdecoded_framesrrrrrrrs     zEncodecModel.decode) output_typer)rrrrrrr?cCs|p |jj}|dkr"t|}|dk r:|dkr:td|dk rR|dkrRtd|dkrv|dkrv||||d\}}|j||||dd}|s||fSt||dS)a Returns: Examples: ```python >>> from datasets import load_dataset >>> from transformers import AutoProcessor, EncodecModel >>> dataset = load_dataset("ashraq/esc50") >>> audio_sample = dataset["train"]["audio"][0]["array"] >>> model_id = "facebook/encodec_24khz" >>> model = EncodecModel.from_pretrained(model_id) >>> processor = AutoProcessor.from_pretrained(model_id) >>> inputs = processor(raw_audio=audio_sample, return_tensors="pt") >>> outputs = model(**inputs) >>> audio_codes = outputs.audio_codes >>> audio_values = outputs.audio_values ```NzBYou specified `audio_codes` but did not specify the `audio_scales`zBYou specified `audio_scales` but did not specify the `audio_codes`F)rr)rr) r:rrrrr1rrr)r9rrrrrrrrrrrTs! zEncodecModel.forward)NNN)N)NN)NNNNN)rrrrr,rrrrWrYrUrrrrrrrrVrrrrrr ENCODEC_INPUTS_DOCSTRINGrr_CONFIG_FOR_DOCrTrZrrr;rrs\  I.  0 r),rrB dataclassesrtypingrrrrrZtorch.utils.checkpointrZmodeling_utilsr r6r r r r rZconfiguration_encodecrZ get_loggerrr2rZ%ENCODEC_PRETRAINED_MODEL_ARCHIVE_LISTrrrModuler r[r^rdrrrrrrrZENCODEC_START_DOCSTRINGrrrrrrsJ        U4!"$,)!