U 9%e@sdZddlZddlmZmZmZddlZddlZddlmZddl m Z m Z m Z ddl mZddlmZmZmZmZdd lmZdd lmZmZmZmZd d lmZeeZd Z dZ!d dgZ"d,ej#ej$ej%e&dddZ'd-ej(ej$ee&dddZ)Gdddej*Z+Gdddej,Z-Gdddej,Z.GdddeZ/dZ0dZ1ed e0Gd!d"d"e/Z2ed#e0Gd$d%d%e/Z3ed&e0Gd'd(d(e/Z4ed)e0Gd*d+d+e/Z5dS).z PyTorch BioGPT model.N)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentions SequenceClassifierOutputWithPastTokenClassifierOutput)PreTrainedModel)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardlogging) BioGptConfigzmicrosoft/biogptrzmicrosoft/BioGPT-Large)input_ids_shapedtypedevicepast_key_values_lengthcCs|\}}tj||ft|j|d}tj|d|d}|||d|ddkd||}|dkrtj tj ||||d|gdd}|ddddddf |d|||S)zB Make causal mask used for bi-directional self-attention. rrrrrdimN) torchfullfinfominarangesizeZ masked_fill_viewtocatZzerosexpand)rrrrbsztgt_lenmaskZ mask_condr,i/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/transformers/models/biogpt/modeling_biogpt.py_make_causal_mask9s"  r.)r+rr*cCsj|\}}|dk r|n|}|ddddddf|d|||}d|}||tjt|jS)z_ Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`. Nr?)r$r(r&Z masked_fillrboolr!r")r+rr*r)src_lenZ expanded_maskZ inverted_maskr,r,r- _expand_maskKs  *r2cs@eZdZdZeedfdd Zd ejedfdd ZZ S) BioGptLearnedPositionalEmbeddingzN This module learns positional embeddings up to a fixed maximum size. )num_embeddings embedding_dimcsd|_t||j|dS)N)offsetsuper__init__)selfr4r5 __class__r,r-r9_sz)BioGptLearnedPositionalEmbedding.__init__r)attention_maskrcsN|}tj|dd||d}|dd|df}t||jS)z3`input_ids_shape` is expected to be [bsz x seqlen].rrN)longrZcumsumtype_asr8forwardr7)r:r=r positionsr;r,r-r@es z(BioGptLearnedPositionalEmbedding.forward)r) __name__ __module__ __qualname____doc__intr9r LongTensorr@ __classcell__r,r,r;r-r3Zsr3c seZdZdZdeeeeedfdd Zej eedd d Z dej e ej e e ej e ej e ej ee ej e ej e e ej fd d dZ ZS)BioGptAttentionz=Multi-headed attention from 'Attention Is All You Need' paperFT) embed_dim num_headsdropout is_decoderbiascst||_||_||_|||_|j||jkrNtd|jd|d|jd|_||_t j |||d|_ t j |||d|_ t j |||d|_ t j |||d|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩rO)r8r9rKrLrMhead_dim ValueErrorscalingrNrLineark_projv_projq_projout_proj)r:rKrLrMrNrOr;r,r-r9vs   zBioGptAttention.__init__)tensorseq_lenr)cCs ||||j|jddS)Nrr6)r%rLrQ transpose contiguous)r:rYrZr)r,r,r-_shapeszBioGptAttention._shapeN) hidden_stateskey_value_statespast_key_valuer=layer_head_maskoutput_attentionsreturncCsx|dk }|\}} } |||j} |r\|dk r\|djd|jdkr\|d} |d} n|r|||d|} |||d|} n|dk r|||d|} |||d|} tj|d| gdd} tj|d| gdd} n(|||d|} |||d|} |j r | | f}||j d|j f}|| | |j |} | j |} | j |} | d}t| | dd}|||j | |fkrtd||j | |fd||dk r ||d| |fkrtd |d| |fd|| ||j | ||}| ||j | |}tjj|dd}|dk r||j fkrhtd |j fd|| dddd| ||j | |}| ||j | |}|r| ||j | |}| ||j | |}nd}tjj||j|jd }t|| }|||j | |j fkr4td ||j | |j fd|| ||j | |j }|dd}| || |j}||}|||fS) z#Input shape: Batch x Time x ChannelNrr6rrrz$Attention weights should be of size z , but is z!Attention mask should be of size z/Head mask for a single layer should be of size ptrainingz `attn_output` should be of size )r$rWrSshaper]rUrVrr'rNrLrQr%ZreshapeZbmmr[rRr functionalZsoftmaxrMrfrKrX)r:r^r_r`r=rarbZis_cross_attentionr)r*_Z query_statesZ key_statesZ value_statesZ proj_shaper1Z attn_weightsZattn_weights_reshapedZ attn_probsZ attn_outputr,r,r-r@s~       "    zBioGptAttention.forward)rJFT)NNNNF)rBrCrDrErFfloatr0r9rTensorr]rrr@rHr,r,r;r-rIss4 rIc sxeZdZedfdd Zd ejeejeejeeejee ee eej eeej ej ffddd Z Z S) BioGptDecoderLayerconfigcst|j|_t|j|j|jdd|_|j|_ t |j |_ |j |_ t|j|_t|j|j|_t|j|j|_t|j|_dS)NT)rKrLrMrN)r8r9 hidden_sizerKrIZnum_attention_headsZattention_probs_dropout_prob self_attnhidden_dropout_probrMr Z hidden_act activation_fnactivation_dropoutr LayerNormself_attn_layer_normrTZintermediate_sizefc1fc2final_layer_normr:rnr;r,r-r9s  zBioGptDecoderLayer.__init__NFT)r^r=rar`rb use_cachercc Cs|}||}|dk r"|ddnd}|j|||||d\}} } tjj||j|jd}||}|}||}||}||}tjj||j |jd}| |}tjj||j|jd}||}|f} |r| | f7} |r| | f7} | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. past_key_value (`Tuple(torch.FloatTensor)`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). Nr6)r^r`r=rarbrd) rurprrhrMrfrxrvrrrsrw) r:r^r=rar`rbrzZresidualZself_attn_past_key_valueZself_attn_weightsZpresent_key_valueoutputsr,r,r-r@"s4        zBioGptDecoderLayer.forward)NNNFT) rBrCrDrr9rrkrrr0 FloatTensorr@rHr,r,r;r-rl s rlc@s.eZdZdZeZdZdZddZd ddZ d S) BioGptPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. biogptTcCst|tjr:|jjjd|jjd|jdk r|jj nft|tj rz|jjjd|jjd|j dk r|jj|j  n&t|tj r|jj |jj ddS)zInitialize the weightsrJ)ZmeanZstdNr/) isinstancerrTweightdataZnormal_rnZinitializer_rangerOZzero_ Embedding padding_idxrtZfill_)r:moduler,r,r- _init_weightsks      z#BioGptPreTrainedModel._init_weightsFcCst|tr||_dSN)r BioGptModelgradient_checkpointing)r:rvaluer,r,r-_set_gradient_checkpointing{s z1BioGptPreTrainedModel._set_gradient_checkpointingN)F) rBrCrDrEr config_classZbase_model_prefixZsupports_gradient_checkpointingrrr,r,r,r-r}as r}aJ This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`~BioGptConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z`The bare BioGPT Model transformer outputting raw hidden-states without any specific head on top.cseZdZedfdd ZddZddZdd Zee d e e e ed deejeejeejeejeeeejeeeeeeeeeee fd ddZZS)rrmcst|_j|_j|_j|_j|_ j rDt jnd|_ tj|j|j |_tj|j|_tfddtjD|_t|j|_d|_|dS)Nr/csg|] }tqSr,)rl).0rirmr,r- sz(BioGptModel.__init__..F)r8r9rn layerdroprqrMrorK pad_token_idrZscale_embeddingmathsqrt embed_scalerr vocab_size embed_tokensr3Zmax_position_embeddingsembed_positionsZ ModuleListrangeZnum_hidden_layerslayersrt layer_normr post_initryr;rmr-r9s  zBioGptModel.__init__cCs|jSrrr:r,r,r-get_input_embeddingssz BioGptModel.get_input_embeddingscCs ||_dSrrr:rr,r,r-set_input_embeddingssz BioGptModel.set_input_embeddingscCs`d}|ddkr$t||j|j|d}|dk r\t||j|dd|j}|dkrT|n||}|S)Nrr)rr)r*)r.rrr2r&)r:r= input_shape inputs_embedsrZcombined_attention_maskZexpanded_attn_maskr,r,r-_prepare_decoder_attention_masks z+BioGptModel._prepare_decoder_attention_maskbatch_size, sequence_length checkpoint output_typerN) input_idsr= head_maskrpast_key_valuesrzrboutput_hidden_states return_dictrcc  sFdk r n|jj|dk r |n|jj}dk r4n|jj| dk rH| n|jj} |dk rj|dk rjtdnN|dk r|} | } n8|dk r|dd} |dddddf} ntd|dk r|ddjdnd} |dkr|| |j }|dkrt j |jddt j |j d}n<|jd| | dkrTtd|jdd | | dd ||| } ||| || }|| }tjj||j|jd }|jr|jrrtd d |rdnd}rdnd}d}rdnd}t|jD]\}}|r||f7}|jr$t g}||jkr$q|dk r6||nd}|jr|jrfdd}t jj|||||dk r|||ndd}n&||||dk r||nd|d}|d}r||rdndf7}r||df7}q|r||f7}||}r|nd}| s4tdd|||||fDSt|||||dS)NzDYou cannot specify both input_ids and inputs_embeds at the same timerz5You have to specify either input_ids or inputs_embedsrr6rrz'The provided attention mask has length z, but its length should be z0 (sum of the lengths of current and past inputs)rdzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fr,csfdd}|S)Ncs|fSrr,)inputs)rrbrzr,r-custom_forwardOszJBioGptModel.forward..create_custom_forward..custom_forwardr,)rrrbrz)rr-create_custom_forwardNsz2BioGptModel.forward..create_custom_forward)r=rar`rbrzcss|]}|dk r|VqdSrr,)rvr,r,r- wsz&BioGptModel.forward..)Zlast_hidden_staterr^ attentionscross_attentions) rnrbrrzuse_return_dictrRr$rgrrrZonesr0rrrrrhrMrfrloggerZ warning_once enumeraterZrandrutilsrrtupler )r:rr=rrrrzrbrrinputrrrAr^Zall_hidden_statesZall_self_attnsZall_cross_attentionsZnext_decoder_cacheidxZ decoder_layerZdropout_probabilityr`rZ layer_outputsZ next_cacher,rr-r@s            zBioGptModel.forward) NNNNNNNNN)rBrCrDrr9rrrrBIOGPT_INPUTS_DOCSTRINGformatr_CHECKPOINT_FOR_DOCr _CONFIG_FOR_DOCrrrGr|rrkr0rr@rHr,r,r;r-rs>  rzHBioGPT Model with a `language modeling` head on top for CLM fine-tuning.cseZdZdgZfddZddZddZee de e e e d deejeejeejeejeeeejeejeeeeeeeeeee fd d d ZdddZeddZZS)BioGptForCausalLMzoutput_projection.weightcs8t|t||_tj|j|jdd|_| dSNFrP) r8r9rr~rrTroroutput_projectionrryr;r,r-r9s  zBioGptForCausalLM.__init__cCs|jSrrrr,r,r-get_output_embeddingssz'BioGptForCausalLM.get_output_embeddingscCs ||_dSrr)r:Znew_embeddingsr,r,r-set_output_embeddingssz'BioGptForCausalLM.set_output_embeddingsrrN) rr=rrrlabelsrzrbrrrcc  Cs| dk r | n|jj} |j|||||||| | d } | d} || } d}|dk r| ddddddf}|ddddf}t}||d|jj|d}| s| f| dd}|dk r|f|S|St|| | j | j | j | j dS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` N)r=rrrrzrbrrrrr)losslogitsrr^rr) rnrr~rr\rr%rr rr^rr)r:rr=rrrrrzrbrrr{Zsequence_outputZprediction_scoresZlm_lossZshifted_prediction_scoresloss_fctoutputr,r,r-r@s>  zBioGptForCausalLM.forwardcKsX|r|dddfd}|dk r4|dkr4d|i}nd|i}||||dd|S)Nrrrrz)r=rrz)Z unsqueezeupdateget)r:rr=rrkwargsZ model_inputsr,r,r-prepare_inputs_for_generations z/BioGptForCausalLM.prepare_inputs_for_generationcs.d}|D] }|tfdd|Df7}q|S)Nr,c3s"|]}|d|jVqdS)rN)Z index_selectr&r)rZ past_statebeam_idxr,r-rsz3BioGptForCausalLM._reorder_cache..)r)rrZreordered_pastZ layer_pastr,rr-_reorder_caches z BioGptForCausalLM._reorder_cache) NNNNNNNNNN)NN)rBrCrDZ_tied_weights_keysr9rrrrrrrr rrrrGr|rrkr0rr@r staticmethodrrHr,r,r;r-rsL   : rz BioGPT Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. cseZdZfddZeeeeee dde e j e e j e e j e e j e eee je e j e e j e ee ee ee eeeefd ddZZS) BioGptForTokenClassificationcsjt||j|_t||_t|dr:|jdk r:|j}n|j}t ||_ t |j |j|_ |dS)Nclassifier_dropout)r8r9 num_labelsrr~hasattrrrqrZDropoutrMrTro classifierr)r:rnrr;r,r-r9s   z%BioGptForTokenClassification.__init__rN) rtoken_type_idsr=rrrrrzrbrrrcc  Cs | dk r | n|jj} |j||||||| | | d } | d} || } || }d}|dk rt}|dk r|ddk}|d|j}t ||dt |j  |}|||}n||d|j|d}| s|f| dd}|dk r|f|S|St ||| j| jdS) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrr=rrrzrbrrrrrr6)rrr^r)rnrr~rMrrr%rrwhererYZ ignore_indexr?rr^r)r:rrr=rrrrrzrbrrtransformer_outputsr^rrrZ active_lossZ active_logitsZ active_labelsrr,r,r-r@sJ    z$BioGptForTokenClassification.forward) NNNNNNNNNNN)rBrCrDr9rrrrrrrrrGr|rrkr0rr@rHr,r,r;r-rs@  ra The BioGpt Model transformer with a sequence classification head on top (linear layer). [`BioGptForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it is required to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). cseZdZedfdd Zeeeee e dd e e j e e je e je eee je e je e j e ee ee ee eeee fd ddZd d Zd d ZZS)BioGptForSequenceClassificationrmcs@t||j|_t||_tj|j|jdd|_| dSr) r8r9rrr~rrTroscorerryr;r,r-r9es   z(BioGptForSequenceClassification.__init__rN) rr=rrrrrzrbrrrcc  Cs | dk r | n|jj} |j|||||||| | d } | d} || } |dk r^|jdd\}}n|jdd\}}|jjdkrd}nD|dk rt||jjdd | j }nd}t |j jd| tj|| j d|f}d}|dk r|jjdkrJ|jdkrd |j_n:|jdkrB|jtjks8|jtjkrBd |j_nd |j_|jjd krt}|jdkr~|||}n |||}nN|jjd krt}||d|j|d}n|jjd krt}|||}| s|f| dd}|dk r|f|S|St||| j| j| jd S) rNrrr6rrz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`rZ regressionZsingle_label_classificationZmulti_label_classification)rrrr^r)rnrr~rrgrrnesumr&rrwarningr<rBr#Z problem_typerrr>rFrZsqueezerr%rr rr^r)r:rr=rrrrrzrbrrrr^rZ batch_sizeZsequence_lengthZ pooled_logitsrrrr,r,r-r@nsn   $    (    z'BioGptForSequenceClassification.forwardcCs|jjSrr~rrr,r,r-rsz4BioGptForSequenceClassification.get_input_embeddingscCs ||j_dSrrrr,r,r-rsz4BioGptForSequenceClassification.set_input_embeddings) NNNNNNNNNN)rBrCrDrr9rrrrr rrrrGr|rrkr0rr@rrrHr,r,r;r-rUs@  Xr)r)N)6rErtypingrrrrZtorch.utils.checkpointrZtorch.nnrrrZ activationsr Zmodeling_outputsr r r rZmodeling_utilsrrrrrrZconfiguration_biogptrZ get_loggerrBrrrZ$BIOGPT_PRETRAINED_MODEL_ARCHIVE_LISTSizerrrFr.rkr2rr3ModulerIrlr}ZBIOGPT_START_DOCSTRINGrrrrrr,r,r,r-sj       T 6AqU