U ,-es@sdZddlZddlmZmZmZmZddlZddl Z ddl m Z ddl m Z ddlmZmZddlmZmZdd lmZmZdd lmZmZmZmZd d lmZd dlmZm Z m!Z!m"Z"m#Z#d dl$m%Z%m&Z&m'Z'e(e)Z*dZ+dZ,ddgZ-Gddde j.Z/Gddde j.Z0Gddde j.Z1Gddde j.Z2GdddeZ3dZ4dZ5dZ6d Z7Gd!d"d"e3Z8ed#e4j9d$d%Gd&d'd'e8Z:ed(e4j9d)d%Gd*d+d+e8Z;ed,e4j9d-d%Gd.d/d/e3Z)r7r?r+r@r;r;r< _merge_headsgs zBarkSelfAttention._merge_headsNc Cst||dddt|j}|jr||d|d}}||j dddd|||d|fdkt |j j }|dk r||}t jj|dd}||j }||}|dk r||}t||} | |fS)Nr=rB?rdim)r3matmulrCmathsqrtr,r1r> masked_fillr finfor"minrrsoftmaxtor') r7querykeyvalueattention_mask head_mask attn_weightsZ query_lengthZ key_length attn_outputr;r;r<_attnss $(    zBarkSelfAttention._attncCs||j|jdd\}}} |||j|j}|||j|j}|| |j|j} |dk r|d} |d} tj| |fdd}tj| | fdd} |dkr|| f} nd} |||| ||\} }| | |j|j} | | } | | } | | f}|r||f7}|S)NrrGrrrBT) r/splitr*rAr+r,r3catrXrEr0r()r7 hidden_statesrTpast_key_valuesrU use_cacheoutput_attentionsrQrRrSZpast_keyZ past_valueZpresentrWrVoutputsr;r;r<forwards(     zBarkSelfAttention.forward)F)NN)NNNFF) __name__ __module__ __qualname__r$rArErXr` __classcell__r;r;r9r<r=s "rcs*eZdZdZdfdd ZddZZS) BarkLayerNormzOLayerNorm but with an optional bias. PyTorch doesn't support simply bias=False.Tcs:ttt||_|r0tt|nd|_dSN) r#r$r Parameterr3r4weightZzerosr )r7r)r r9r;r<r$s zBarkLayerNorm.__init__cCstj||jj|j|jddS)Ngh㈵>)Zeps)FZ layer_normrhshaper )r7inputr;r;r<r`szBarkLayerNorm.forward)T)rarbrc__doc__r$r`rdr;r;r9r<resrecs$eZdZfddZddZZS)BarkMLPcs^ttj|jd|j|jd|_tjd|j|j|jd|_t|j |_ t |_ dS)Nr) r#r$rr.r)r in_projr0r&r%ZGELUgelur7r8r9r;r<r$s  zBarkMLP.__init__cCs,||}||}||}||}|Srf)rorpr0r%)r7r[r;r;r<r`s     zBarkMLP.forwardrarbrcr$r`rdr;r;r9r<rms rmcs(eZdZdfdd ZdddZZS) BarkBlockFcslt|r4t|j|jd|_t|j|jd|_nt|j|_t|j|_t ||d|_ t ||_ dS)Nrr1) r#r$rer)r layernorm_1 layernorm_2r LayerNormrattnrmmlp)r7r8r1r9r;r<r$s zBarkBlock.__init__Nc Csv||}|j||||||d}|d} |dd} || }||||}|r`|f| } n|f| dd} | S)Nr\rTrUr]r^rr)rurxryrv) r7r[r\rTrUr]r^Zintermediary_hidden_statesZ attn_outputsrWr_r;r;r<r`s&    zBarkBlock.forward)F)NNNFFrrr;r;r9r<rssrscsNeZdZdZeZdZddZfddZe e j ddd Z d d d Z Z S) BarkPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. FcCst|tjfr<|jjjd|jjd|jdk r|jj nft|tj r||jjjd|jjd|j dk r|jj|j  n&t|tj r|jj |jj ddS)zInitialize the weights.g)ZmeanZstdNrF) isinstancerr.rhdataZnormal_r8Zinitializer_ranger Zzero_ EmbeddingZ padding_idxrwZfill_r7moduler;r;r< _init_weightss     z!BarkPreTrainedModel._init_weightscstj||dSrf)r#r$)r7inputskwargsr9r;r<r$%szBarkPreTrainedModel.__init__returncCs\t|dst|S|D]8}t|drt|jdr|jjdk rt|jjSqt|Sz `torch.device`: The device on which the module is (assuming that all the module parameters are on the same device). _hf_hookexecution_deviceN)hasattrrmodulesrrr3devicerr;r;r<r(s    zBarkPreTrainedModel.devicecCs(t|tst|tst|tr$||_dSrf)r|BarkCausalModel BarkFineModel BarkModelgradient_checkpointing)r7rrSr;r;r<_set_gradient_checkpointing<sz/BarkPreTrainedModel._set_gradient_checkpointing)F)rarbrcrlr config_classZsupports_gradient_checkpointingrr$propertyr3rrrdr;r;r9r<r{ s r{aG This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`{config}`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. aI This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`BarkConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: codebook_idx (`int`): Index of the codebook that will be predicted. input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, number_of_codebooks)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Initially, indices of the first two codebooks are obtained from the `coarse` sub-model. The rest is predicted recursively by attending the previously predicted channels. The model predicts on windows of length 1024. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.Tensor` of shape `(encoder_layers, encoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): NOT IMPLEMENTED YET. input_embeds (`torch.FloatTensor` of shape `(batch_size, input_sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. If `past_key_values` is used, optionally only the last `input_embeds` have to be input (see `past_key_values`). This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `input_ids` of shape `(batch_size, sequence_length)`. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.Tensor` of shape `(encoder_layers, encoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. input_embeds (`torch.FloatTensor` of shape `(batch_size, input_sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. Here, due to `Bark` particularities, if `past_key_values` is used, `input_embeds` will be ignored and you have to use `input_ids`. If `past_key_values` is not used and `use_cache` is set to `True`, `input_embeds` is used in priority instead of `input_ids`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. cseZdZeZfddZddZddZddd Ze e de e j e ee je e j e e j e e j e e je e j e ee ee ee eeee j efd d d Zeeee j e j eee j d ddZZS)rcst|_tjj|_tjj|_ t j |_ t fddtjD|_tjjd|_tjjjdd|_d|_|dS)Ncsg|]}tddqS)Trtrs.0_r8r;r< sz,BarkCausalModel.__init__..rF)r#r$r8rr~input_vocab_sizer)input_embeds_layerr2position_embeds_layerr&r%drop ModuleListrange num_layerslayersrer layernorm_finalr.output_vocab_sizelm_headr post_initrqr9rr<r$s  zBarkCausalModel.__init__cCs|jSrfrr7r;r;r<get_input_embeddingssz$BarkCausalModel.get_input_embeddingscCs ||_dSrfrr7Znew_embeddingsr;r;r<set_input_embeddingssz$BarkCausalModel.set_input_embeddingsNcKsD|dd}|dd}|dd}|dk rN|jd}|dddgf}d}n(|dk rl|drl|jd}n |jd}|dk r|ddd|f}|dk r|ddd|f}|dk r|dkr|dd}||dkd|r|dddfd}nd}|dk r.|dr.d|||d||dS|||d||d S) N input_embedsrT position_idsrr=r]r) input_idsrr\r]rrT)rr\r]rrT)getrjlongZcumsum masked_fill_ unsqueeze)r7rr\rrrTrZseq_lenr;r;r<prepare_inputs_for_generationsD       z-BarkCausalModel.prepare_inputs_for_generation) rr\rTrrUlabelsrr]r^output_hidden_states return_dictrc  sZdk r n|jj| dk r | n|jj} dk r4n|jj| dk rH| n|jj} |dk rj|dk rjtdn8|dk r||dkr|n&|dk r||}n|dk rntd|dd} |jd} | d}|dk r|j n|j }|dkrd}t dgt |j }n|ddd}|dkr8t j|||t j|d}|d}||}|dk r| dkr^td|| d}|ddddddf}|j|jd}d |t |jj}|||jj}|||}| |df}|jr|jrrtd d rd nd}rd nd}| rd nd}tt|j |D]\}\}}| rN||f}|jr|jrfd d}t j j!!|||d|||}n||||||d}|d}r||df}r2||rdndf}q2|"|}||}| r||f}|#|}d}|dk r&t$d| sHt ddd||||fDSt%|||||dS)NCYou cannot specify both input_ids and input_embeds at the same time4You have to specify either input_ids or input_embedsr=rrBr"r$batch_size has to be defined and > 0r!rFzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fr;csfdd}|S)Ncs|fSrfr;)r)rr^r]r;r<custom_forwardyszNBarkCausalModel.forward..create_custom_forward..custom_forwardr;)rrr^r])rr<create_custom_forwardxsz6BarkCausalModel.forward..create_custom_forwardrzrrzXTraining is not implemented yet for Bark - ensure you do not pass `labels` to the model.css|]}|dk r|VqdSrfr;rvr;r;r< sz*BarkCausalModel.forward..)losslogitsr\r[ attentions)&r8r^rr]use_return_dictr-rr>rjrtuplelenrr3arangerrrr6rPr"rMrN get_head_maskrrrZtrainingloggerZ warning_once enumerateziputils checkpointrrNotImplementedErrorr )r7rr\rTrrUrrr]r^rr input_shape batch_size seq_lengthrZ past_lengthposition_embedsr[ output_shapeZpresent_key_valuesall_self_attentionsall_hidden_statesiblockZpast_layer_key_valuesrr_rrr;rr<r`s                 zBarkCausalModel.forward)r\beam_idxrcstfdd|DS)a This function is used to re-order the `past_key_values` cache if [`~PreTrainedModel.beam_search`] or [`~PreTrainedModel.beam_sample`] is called. This is required to match `past_key_values` with the correct beam_idx at every generation step. c3s$|]}tfdd|DVqdS)c3s"|]}|d|jVqdS)rN)Z index_selectrPr)rZ past_staterr;r<rsz;BarkCausalModel._reorder_cache...Nr)rZ layer_pastrr;r<rsz1BarkCausalModel._reorder_cache..r)r\rr;rr<_reorder_caches zBarkCausalModel._reorder_cache)N) NNNNNNNNNNN)rarbrcrrr$rrrr"BARK_CAUSAL_MODEL_INPUTS_DOCSTRINGrr3TensorrZ FloatTensor LongTensorr5rr r` staticmethodrrdr;r;r9r<rsJ  4 "  rzBark semantic (or text) model. It shares the same architecture as the coarse model. It is a GPT-2 like autoregressive model with a language modeling head on top.rrcsLeZdZdZeZdejee e e ejfe ejej dfdd Z ZS)BarkSemanticModelsemanticN)rsemantic_generation_confighistory_promptrTrc s|dkrtd|jd}|j}||j}|dk rH|d||j}|dk r|d| d}tjj |d|t |f|j dd}n t j |j g|t jd|j}t j|d|dd }t j |jgg|t jd|j} t j||ddd|f||ddd|df|| gdd } tt|j|j } | tt|j d|jjt| } tjt j||dft jd|jf| | g|d |} | dd|ddf} | S) a Generates text semantic tokens from an input prompt and an additional optional `Bark` speaker prompt. Args: input_ids (`Optional[torch.Tensor]` of shape (batch_size, seq_len), *optional*): Input ids, i.e tokenized input sentences. Will be truncated up to semantic_generation_config.max_input_semantic_length tokens. Note that the output audios will be as long as the longest generation among the batch. semantic_generation_config (`BarkSemanticGenerationConfig`): Generation config indicating how to generate the semantic tokens. history_prompt (`Optional[Dict[str,torch.Tensor]]`, *optional*): Optional `Bark` speaker prompt. attention_mask (`Optional[torch.Tensor]`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) Returns: torch.LongTensor: Output semantic tokens. N/`semantic_generation_config` has to be providedrrsemantic_promptconstant)rSmoder!rG)rlogits_processorgeneration_config)r-rjmax_input_semantic_lengthZtext_encoding_offsetrLr5Ztext_pad_tokenrrpadrsemantic_pad_tokenr3r?intrPrrepeat_interleaveZsemantic_infer_tokenrZrlistrsemantic_vocab_sizeextendr8rr r#generater4)r7rrrrTrrrZsemantic_historyZ infer_arrayrZtokens_to_suppressZ suppress_tokens_logits_processorsemantic_outputr9r;r<rsl       zBarkSemanticModel.generate)NNN)rarbrcbase_model_prefixrrr3rrrrstrrrrdr;r;r9r<rsrzBark coarse acoustics model. It shares the same architecture as the semantic (or text) model. It is a GPT-2 like autoregressive model with a language modeling head on top.rc speZdZdZeZd eeeeeeee e j fdddZ d e j e eeeee e j fe jdfdd ZZS) BarkCoarseModelcoarse_acousticsN)max_coarse_historysemantic_to_coarse_ratiorr codebook_sizerc Cs|dk r.csg|]}tddqS)Frtrrrr;r<rscs g|]}tjjjddqS)Fr)rr.r)rrrr;r<rsF)r#r$r8rrr n_codes_totalinput_embeds_layersr~r2r)rr&r%rrrrwr n_codes_givenlm_headsrrrqr9rr<r$ s"    zBarkFineModel.__init__cCs|jSrfr rr;r;r<r(sz"BarkFineModel.get_input_embeddingscCs ||_dSrfrrr;r;r<r,sz"BarkFineModel.set_input_embeddingscCs|jSrfrrr;r;r<get_output_embeddings0sz#BarkFineModel.get_output_embeddingscCs ||_dSrfr)r7Znew_output_embeddingsr;r;r<set_output_embeddings4sz#BarkFineModel.set_output_embeddingsNcs}tfdd|D}||djjddk r~jjs~}tfdd|D} |S)Ncsg|]}|qSr;)Z_get_resized_embeddings)rZold_embeddingsnew_num_tokenspad_to_multiple_ofr7r;r<r;sz:BarkFineModel._resize_token_embeddings..rcsg|]}|qSr;)Z_get_resized_lm_head)rZ old_lm_head)rr7r;r<rGs) rrrrrhrjrr8tie_word_embeddingsr)r7rrZold_embeddings_listZnew_embeddings_listZold_lm_head_listZnew_lm_head_listr;rr<_resize_token_embeddings8s  z&BarkFineModel._resize_token_embeddings)rrrcCsx|||}|dkr |dkr |S|djjd|j_|djjd|j_|djjd|_|djjd|_||S)aX Resizes input token embeddings matrix of the model if `new_num_tokens != config.vocab_size`. Takes care of tying weights embeddings afterwards if the model class has a `tie_weights()` method. Arguments: new_num_tokens (`int`, *optional*): The number of new tokens in the embedding matrix. Increasing the size will add newly initialized vectors at the end. Reducing the size will remove vectors from the end. If not provided or `None`, just returns a pointer to the input tokens `torch.nn.Embedding` module of the model without doing anything. pad_to_multiple_of (`int`, *optional*): If set will pad the embedding matrix to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta), or on TPUs which benefit from having sequence lengths be a multiple of 128. For more details about this, or help on choosing the correct value for resizing, refer to this guide: https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tc Return: `torch.nn.Embedding`: Pointer to the input tokens Embeddings Module of the model. Nr)rrhrjr8rZ vocab_size tie_weights)r7rrZ model_embedsr;r;r<resize_token_embeddingsMs z%BarkFineModel.resize_token_embeddingscCst|jddrjg|_|}|}t|jj|jjD]0}|||||d|j d|dq8| D]}t |drr| qrdS)z Tie the weights between the input embeddings list and the output embeddings list. If the `torchscript` flag is set in the configuration, can't handle parameter sharing so we are cloning the weights instead. rTrz lm_heads.z.weight _tie_weightsN) getattrr8Z_tied_weights_keysrrrr rZ_tie_or_clone_weightsappendrrr)r7Zoutput_embeddingsZinput_embeddingsrrr;r;r<rts  zBarkFineModel.tie_weights) r rrTrrUrrr^rrrc s|dk r |n|jj}| dk r | n|jj} | dk r4| n|jj} |dkrLtddk rd|dk rdtddkr||dkr|tddk rևfddt|jD}tj|dd}|ddddddd|d fj dd}| dd} |j d} | d } dk rj n|j }|dkr6tj d| tj|d }|d}||}|dk r| dkr\td || d}|ddddddf}|j|jd }d |t|jj}|||jj}|||}| | df}|rdnd}| rdnd}t|jD]J\}}| r||f}||||||d}|d}|r||d f}q||}||}| rh||f}|j||jj|}d}|dk rtd| stddd|||fDSt||||dS)NrzRCannot predict 0th codebook - 0th codebook should be predicted by the coarse modelrrcs0g|](\}}|dddd|fdqS)Nr=)r)rrrrr;r<rsz)BarkFineModel.forward..r=rGrrrr!rFr;)rTrUr^zTraining is not implemented yetcss|]}|dk r|VqdSrfr;rr;r;r<rsz(BarkFineModel.forward..)rrr[r) r8r^rrr-rr r3rZsumr>rjrrrrrr6rPr"rMrNrrrrrrrrrr )r7r rrTrrUrrr^rrrrrrrr[rrrrrr_rrr;rr<r`s ,            zBarkFineModel.forwardr)r rrfine_generation_configrrrc Ks|dkrtd|dkr td|dkr0td|d|j}|j} |j} ||jdd|j}t ||j |}|jd} |dk rtj |dj d| dd } nd} |j} t |d|j| fd |}| dk rtj| dd| dddf|gd d }| dd| dddfjd }nd}d}|jd | kr\| |jd }t j|ddd|fd |d }|jd | || }tt|}td|d }t|D]}t|| |jd | g}t||| |jd | g}||}|dd||| ddf}t| |jD]}|||j}|dks,|d krT|dd|dd|f}t|d}n`|ddddd|f|}t j|dd dd|| f}|d|f}tj|d d| d}|tj}||dd|d|f<~~qt| |jD]6}|dd|d|f|dd||| ||f<q~q|d ddddd|df}|dkrp|ddddd| f}|jd|jdkrtd|S)ap Generates fine acoustics tokens from input coarse acoustics tokens and an additional optional `Bark` speaker prompt. Args: coarse_output (`torch.Tensor` of shape (batch_size, seq_len)): Input coarse acoustics ids, i.e the output of `BarkCoarseModel.generate`. semantic_generation_config (`BarkSemanticGenerationConfig`): Generation config indicating how to generate the semantic tokens. coarse_generation_config (`BarkCoarseGenerationConfig`): Generation config indicating how to generate the coarse tokens. fine_generation_config (`BarkFineGenerationConfig`): Generation config indicating how to generate the fine tokens. codebook_size (`int`, *optional*, defaults to 1024): Codebook channel size, i.e. the size of the output vocabulary per codebook channel. history_prompt (`Optional[Dict[str,torch.Tensor]]`, *optional*): Optional `Bark` speaker prompt. Returns: torch.LongTensor: Output fine acoustics tokens. Nrrz+`fine_generation_config` has to be provided temperaturerr=Z fine_promptrGrr)rrSrF)Z num_samplesrrBz-input and output should have the same seq_len) r-rr!max_fine_history_lengthmax_fine_input_lengthr6rjrr3 remainderrrTrirZn_fine_codebooksrZrrrrrrNr`rZargmaxrOZreshapeZ multinomialrPZint32rC)r7r rrr rrrr!r"r#rZx_fine_historyZn_coarseZ fine_inputZ n_historyZn_remove_from_endZn_loopsZn_outerZ start_idxZstart_fill_idxZrel_start_fill_idxZ input_bufferZn_innerrZrelevant_logitsZcodebook_predsZprobsr;r;r<rs   *$  " zBarkFineModel.generate)N)NN) NNNNNNNNN)NNNrN)$rarbrcrrrZmain_input_namer$rrrrrrrrr~rrrBARK_FINE_INPUTS_DOCSTRINGr3rrr5rrr r`rrrrrrrdr;r;r9r<rsl   'ora6 The full Bark model, a text-to-speech model composed of 4 sub-models: - [`BarkSemanticModel`] (also referred to as the 'text' model): a causal auto-regressive transformer model that takes as input tokenized text, and predicts semantic text tokens that capture the meaning of the text. - [`BarkCoarseModel`] (also refered to as the 'coarse acoustics' model), also a causal autoregressive transformer, that takes into input the results of the last model. It aims at regressing the first two audio codebooks necessary to `encodec`. - [`BarkFineModel`] (the 'fine acoustics' model), this time a non-causal autoencoder transformer, which iteratively predicts the last codebooks based on the sum of the previous codebooks embeddings. - having predicted all the codebook channels from the [`EncodecModel`], Bark uses it to decode the output audio array. It should be noted that each of the first three modules can support conditional speaker embeddings to condition the output sound according to specific predefined voice. cseZdZeZfddZeejdddZde e ddd Z d d Z e de eje eeejfejd ddZZS)rcsHt|t|j|_t|j|_t|j |_ t |j |_||_dSrf)r#r$rsemantic_configrrcoarse_acoustics_configrrfine_acoustics_configr r from_configZ codec_config codec_modelr8rqr9r;r<r$s     zBarkModel.__init__rcCs\t|jdst|S|jD]8}t|drt|jdr|jjdk rt|jjSqdSr)rrrrrrr3rrr;r;r<rs   zBarkModel.devicer)gpu_idcCstrddlm}ntdtd|}|jjdkrL|dtj ||j j |\|j _ }d}|j |j |j fD]}||||d\}}qv||_||j||d\}}||_dS)a Offloads all sub-models to CPU using accelerate, reducing memory usage with a low impact on performance. This method moves one whole sub-model at a time to the GPU when it is used, and the sub-model remains in GPU until the next sub-model runs. Args: gpu_id (`int`, *optional*, defaults to 0): GPU id on which the sub-models will be loaded and offloaded. r)cpu_offload_with_hookz1`enable_model_cpu_offload` requires `accelerate`.zcuda:cpuN)Zprev_module_hook)rZ accelerater- ImportErrorr3rtyperPcudaZ empty_cacherrrr fine_acoustics_hookr+codec_model_hook)r7r,r-rrhookZcpu_offloaded_modelr;r;r<enable_cpu_offloads"    zBarkModel.enable_cpu_offloadcCs4|dd}|jj|}|j|}|d}|S)z:Turn quantized audio codes into audio array using encodec.rr)rCr+Z quantizerdecodedecoderZsqueeze)r7Z fine_outputZemboutZ audio_arrr;r;r< codec_decodes    zBarkModel.codec_decodeN)rrrcKstf|jj}tf|jj}tf|jj}d|ddi}i}i} |D]\} } | drv| t dd} | || <qJ| dr| t dd} | || <qJ| dr| t dd} | | | <qJ| |kr| || <| |kr| || <| | krJ| | | <qJ|j j |f||d|} |j j | f||||jjd|} |jj | f|||||jjd| }t|d ddk r||j|j|j|_||}t|d ddk r|j|S) aB Generates audio from an input prompt and an additional optional `Bark` speaker prompt. Args: input_ids (`Optional[torch.Tensor]` of shape (batch_size, seq_len), *optional*): Input ids. Will be truncated up to 256 tokens. Note that the output audios will be as long as the longest generation among the batch. history_prompt (`Optional[Dict[str,torch.Tensor]]`, *optional*): Optional `Bark` speaker prompt. Note that for now, this model takes only one speaker prompt per batch. kwargs (*optional*): Remaining dictionary of keyword arguments. Keyword arguments are of two types: - Without a prefix, they will be entered as `**kwargs` for the `generate` method of each sub-model. - With a *semantic_*, *coarse_*, *fine_* prefix, they will be input for the `generate` method of the semantic, coarse and fine respectively. It has the priority over the keywords without a prefix. This means you can, for example, specify a generation strategy for all sub-models except one. Returns: torch.LongTensor: Output generated audio. Example: ```python >>> from transformers import AutoProcessor, BarkModel >>> processor = AutoProcessor.from_pretrained("suno/bark-small") >>> model = BarkModel.from_pretrained("suno/bark-small") >>> # To add a voice preset, you can pass `voice_preset` to `BarkProcessor.__call__(...)` >>> voice_preset = "v2/en_speaker_6" >>> inputs = processor("Hello, my dog is cute, I need him in my life", voice_preset=voice_preset) >>> audio_array = model.generate(**inputs, semantic_max_new_tokens=100) >>> audio_array = audio_array.cpu().numpy().squeeze() ``` rTNZ semantic_Zcoarse_Zfine_)rr)rrrr)rrrr rr2r3)rrr'rr(rr)popitems startswithrrrrrr rr2Zoffloadr+rPrr9r3)r7rrrrrr Zkwargs_semanticZ kwargs_coarseZ kwargs_finerRrSrr outputZaudior;r;r<rsx-             zBarkModel.generate)r)NN)rarbrcrrr$rr3rrrr5r9Zno_gradrrrrrrdr;r;r9r<rs ' r)>rlrJtypingrrrrnumpyrr3rZtorch.nnrriZgeneration.logits_processr r Zmodeling_outputsr r Zmodeling_utilsr rrrrrrautorZconfiguration_barkrrrrrZgeneration_configuration_barkrrrZ get_loggerrarZ_CHECKPOINT_FOR_DOCZ_CONFIG_FOR_DOCZ"BARK_PRETRAINED_MODEL_ARCHIVE_LISTModulerrermrsr{ZBARK_MODEL_START_DOCSTRINGZBARK_START_DOCSTRINGr&rrformatrrrrr;r;r;r<sl    } 65*5 c N |