U 9%e@sdZddlZddlZddlmZddlmZmZmZddl Z ddl Z ddl m Z ddl m Z mZmZddlmZmZdd lmZmZmZmZmZmZdd lmZmZdd lmZmZm Z dd l!m"Z"m#Z#m$Z$m%Z%d dl&m'Z'e%(e)Z*dZ+dZ,dddgZ-ddZ.Gddde j/Z0GdddeZ1Gddde j/Z2Gddde j/Z3Gddde j/Z4Gdd d e j/Z5Gd!d"d"e j/Z6Gd#d$d$e j/Z7Gd%d&d&e j/Z8Gd'd(d(e j/Z9Gd)d*d*e j/Z:Gd+d,d,e j/Z;d-ZGd2d3d3e j/Z?e#d4e<Gd5d6d6e1Z@Gd7d8d8e j/ZAe#d9e<Gd:d;d;e1ZBe#dd>e1ZCe#d?e<Gd@dAdAe1ZDe#dBe<GdCdDdDe1ZEdS)Ez PyTorch ConvBERT model.N) attrgetter)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FNget_activation)"BaseModelOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModelSequenceSummary)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardlogging)ConvBertConfigzYituTech/conv-bert-baserzYituTech/conv-bert-medium-smallzYituTech/conv-bert-smallcCsz ddl}Wn tk r,tdYnXtj|}td||j |}i}|D]4\}}td|d||j ||} | ||<q^ddd d d d d d} |j dkrd} nd} t |j D]D} d| d| d| d<d| d| d| d<d| d| d| d<d| d| d| d<d| d| d| d<d| d| d| d<d| d | d| d!<d| d"| d| d#<d| d$| d| d%<d| d&| d| d'<d| d(| d| d)<d| d*| d| d+<d| d,| d| d-<d| d.| d| d/<d| d0| d| d1<d| d2| d| d3<d| d4| d| d5<d| d6| d7| d| d8<d| d6| d9| d| d:<d| d;| d7| d| d<<d| d;| d9| d| d=<d| d>| d| d?<d| d@| d| dA<q|D]} | d}t|}||}| |}t||}tdB|dC|dD|d7r|dEs|dFs|j}|dGr|ddHd}|dIr|dHdd}|dJr|dK}||_q|S)Lz'Load tf checkpoints in a pytorch model.rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z&Converting TensorFlow checkpoint from zLoading TF weight z with shape z"electra/embeddings/word_embeddingsz&electra/embeddings/position_embeddingsz(electra/embeddings/token_type_embeddingsz"electra/embeddings/LayerNorm/gammaz!electra/embeddings/LayerNorm/betaz!electra/embeddings_project/kernelzelectra/embeddings_project/bias)z!embeddings.word_embeddings.weightz%embeddings.position_embeddings.weightz'embeddings.token_type_embeddings.weightzembeddings.LayerNorm.weightzembeddings.LayerNorm.biaszembeddings_project.weightzembeddings_project.biasrZg_densedensezelectra/encoder/layer_z/attention/self/query/kernelzencoder.layer.z.attention.self.query.weightz/attention/self/query/biasz.attention.self.query.biasz/attention/self/key/kernelz.attention.self.key.weightz/attention/self/key/biasz.attention.self.key.biasz/attention/self/value/kernelz.attention.self.value.weightz/attention/self/value/biasz.attention.self.value.biasz./attention/self/conv_attn_key/depthwise_kernelz4.attention.self.key_conv_attn_layer.depthwise.weightz./attention/self/conv_attn_key/pointwise_kernelz4.attention.self.key_conv_attn_layer.pointwise.weightz"/attention/self/conv_attn_key/biasz(.attention.self.key_conv_attn_layer.biasz'/attention/self/conv_attn_kernel/kernelz(.attention.self.conv_kernel_layer.weightz%/attention/self/conv_attn_kernel/biasz&.attention.self.conv_kernel_layer.biasz&/attention/self/conv_attn_point/kernelz%.attention.self.conv_out_layer.weightz$/attention/self/conv_attn_point/biasz#.attention.self.conv_out_layer.biasz/attention/output/dense/kernelz.attention.output.dense.weightz!/attention/output/LayerNorm/gammaz".attention.output.LayerNorm.weightz/attention/output/dense/biasz.attention.output.dense.biasz /attention/output/LayerNorm/betaz .attention.output.LayerNorm.biasz/intermediate/z/kernelz.intermediate.dense.weightz/biasz.intermediate.dense.biasz/output/z.output.dense.weightz.output.dense.biasz/output/LayerNorm/gammaz.output.LayerNorm.weightz/output/LayerNorm/betaz.output.LayerNorm.biaszTF: z, PT:  z/intermediate/g_dense/kernelz/output/g_dense/kernelz/depthwise_kernelz/pointwise_kernelz/conv_attn_key/bias)Z tensorflow ImportErrorloggererrorospathabspathinfotrainZlist_variablesZ load_variable num_groupsrangenum_hidden_layersZnamed_parametersrtorchZ from_numpyendswithTpermute unsqueezedata)modelconfigZtf_checkpoint_pathtfZtf_pathZ init_varsZtf_datanameshapearrayZ param_mappingZgroup_dense_namejparam param_nameZ retrieverresultZtf_namevaluer>m/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/transformers/models/convbert/modeling_convbert.pyload_tf_weights_in_convbert8s                                                       r@csReZdZdZfddZdeejeejeejeejejdddZ Z S) ConvBertEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.csttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_|jdt|jddd|jdtj|jtjddddS) N) padding_idxZeps position_ids)rr!F) persistenttoken_type_ids)dtype)super__init__r Embedding vocab_sizeembedding_sizeZ pad_token_idword_embeddingsZmax_position_embeddingsposition_embeddingsZtype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutZregister_bufferr-ZarangeexpandzerosrDsizelongselfr4 __class__r>r?rIs  zConvBertEmbeddings.__init__N) input_idsrFrD inputs_embedsreturnc Cs|dk r|}n|dd}|d}|dkrH|jddd|f}|dkrt|dr|jddd|f}||d|}|}ntj|tj|jjd}|dkr| |}| |} | |} || | } | | } | | } | S)Nr!rrFrrGdevice)rWrDhasattrrFrUr-rVrXrarMrNrOrPrT) rZr]rFrDr^ input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedrNrO embeddingsr>r>r?forwards(        zConvBertEmbeddings.forward)NNNN) __name__ __module__ __qualname____doc__rIrr- LongTensor FloatTensorrh __classcell__r>r>r[r?rAs rAc@s2eZdZdZeZeZdZdZ ddZ d ddZ d S) ConvBertPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. convbertTcCst|tjr:|jjjd|jjd|jdk r|jj nft|tj rz|jjjd|jjd|j dk r|jj|j  n&t|tj r|jj |jj ddS)zInitialize the weightsZmeanZstdNg?) isinstancerLinearweightr2normal_r4initializer_rangebiasZzero_rJrBrPZfill_)rZmoduler>r>r? _init_weightss      z%ConvBertPreTrainedModel._init_weightsFcCst|tr||_dSN)rtConvBertEncodergradient_checkpointing)rZrzr=r>r>r?_set_gradient_checkpointing s z3ConvBertPreTrainedModel._set_gradient_checkpointingN)F) rirjrkrlr config_classr@Zload_tf_weightsZbase_model_prefixZsupports_gradient_checkpointingr{rr>r>r>r?rpsrpcs4eZdZdZfddZejejdddZZS)SeparableConv1DzSThis class implements separable convolution, i.e. a depthwise and a pointwise layerc s~ttj|||||ddd|_tj||ddd|_tt|d|_ |jj j j d|j d|jj j j d|j ddS)Nr F) kernel_sizegroupspaddingryr)rryrrrs)rHrIrZConv1d depthwise pointwise Parameterr-rVryrvr2rwrx)rZr4Z input_filtersZoutput_filtersrkwargsr[r>r?rIs zSeparableConv1D.__init__ hidden_statesr_cCs"||}||}||j7}|Sr|)rrry)rZrxr>r>r?rh#s   zSeparableConv1D.forward rirjrkrlrIr-Tensorrhror>r>r[r?rs rc sfeZdZfddZddZd ejeejeejeejee e ejeejfddd Z Z S) ConvBertSelfAttentioncs`t|j|jdkr>t|ds>td|jd|jd|j|j}|dkrb|j|_d|_n||_|j|_|j|_|j|jdkrtd|j|jd|_|j|j|_ t |j|j |_ t |j|j |_ t |j|j |_t||j|j |j|_t |j |j|j|_t |j|j |_t j|jdgt|jdddgd |_t |j|_dS) NrrLzThe hidden size (z6) is not a multiple of the number of attention heads ()rz6hidden_size should be divisible by num_attention_headsr )rr)rHrI hidden_sizenum_attention_headsrb ValueErrorZ head_ratioconv_kernel_sizeattention_head_size all_head_sizerruquerykeyr=rkey_conv_attn_layerconv_kernel_layerconv_out_layerZUnfoldintunfoldrRZattention_probs_dropout_probrT)rZr4Znew_num_attention_headsr[r>r?rI+s@  zConvBertSelfAttention.__init__cCs6|dd|j|jf}|j|}|ddddS)Nr!rr rr )rWrrviewr0)rZrZ new_x_shaper>r>r?transpose_for_scoresRs z*ConvBertSelfAttention.transpose_for_scoresNFrattention_mask head_maskencoder_hidden_statesoutput_attentionsr_cCsV||}|d}|dk r2||}||} n||}||} ||dd} | dd} ||} ||} || } t| |}| |}t |d|j dg}tj |dd}| |}t ||d|jg}|ddd}tjj||j dgd|j dddgdd}|dd |d|j|j }t |d|j|j g}t||}t |d|jg}t| | dd}|t|j}|dk r||}tjj |dd}||}|dk r||}t|| }|dddd}t ||d|j|jg}t||gd}|dd|j|jdf}|j|}|rL||fn|f}|S) Nrrr r!dim)rZdilationrZstrider )rrWrr=rZ transposerr-multiplyrreshaperZsoftmaxrr contiguousr1rZ functionalrrmatmulmathsqrtrTr0rcatr)rZrrrrrZmixed_query_layer batch_sizeZmixed_key_layerZmixed_value_layerZmixed_key_conv_attn_layerZ query_layerZ key_layerZ value_layerZconv_attn_layerrrZattention_scoresZattention_probsZ context_layerZconv_outZnew_context_layer_shapeoutputsr>r>r?rhWsj                    zConvBertSelfAttention.forward)NNNF) rirjrkrIrr-rrrnboolrrhror>r>r[r?r*s 'rcs4eZdZfddZejejejdddZZS)ConvBertSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNrC) rHrIrrurrrPrQrRrSrTrYr[r>r?rIs zConvBertSelfOutput.__init__r input_tensorr_cCs&||}||}|||}|Sr|rrTrPrZrrr>r>r?rhs  zConvBertSelfOutput.forwardrirjrkrIr-rrhror>r>r[r?rs rc sfeZdZfddZddZd ejeejeejeejee e ejeejfddd Z Z S) ConvBertAttentioncs*tt||_t||_t|_dSr|)rHrIrrZroutputset pruned_headsrYr[r>r?rIs   zConvBertAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr)lenrrZrrrrrrr=rrrunion)rZheadsindexr>r>r? prune_headss zConvBertAttention.prune_headsNFrc Cs8||||||}||d|}|f|dd}|S)Nrr)rZr) rZrrrrrZ self_outputsattention_outputrr>r>r?rhszConvBertAttention.forward)NNNF) rirjrkrIrr-rrrnrrrhror>r>r[r?rs rcs0eZdZfddZejejdddZZS)GroupedLinearLayercsjt||_||_||_|j|j|_|j|j|_tt |j|j|j|_ tt ||_ dSr|) rHrI input_size output_sizer* group_in_dimZ group_out_dimrrr-emptyrvry)rZrrr*r[r>r?rIs zGroupedLinearLayer.__init__rcCsrt|d}t|d|j|jg}|ddd}t||j}|ddd}t||d|j g}||j }|S)Nrr!rr ) listrWr-rr*rr0rrvrry)rZrrrr>r>r?rhs zGroupedLinearLayer.forwardrr>r>r[r?rs rcs0eZdZfddZejejdddZZS)ConvBertIntermediatecsdt|jdkr(t|j|j|_nt|j|j|jd|_t |j t rXt |j |_ n|j |_ dS)Nrrrr*)rHrIr*rrurintermediate_sizerrrt hidden_actstrr intermediate_act_fnrYr[r>r?rIs   zConvBertIntermediate.__init__rcCs||}||}|Sr|)rrrZrr>r>r?rh s  zConvBertIntermediate.forwardrr>r>r[r?rs rcs4eZdZfddZejejejdddZZS)ConvBertOutputcsdt|jdkr(t|j|j|_nt|j|j|jd|_tj |j|j d|_ t |j |_ dS)NrrrC)rHrIr*rrurrrrrPrQrRrSrTrYr[r>r?rIs  zConvBertOutput.__init__rcCs&||}||}|||}|Sr|rrr>r>r?rhs  zConvBertOutput.forwardrr>r>r[r?rs rc sneZdZfddZd ejeejeejeejeejeee ejeejfdddZ dd Z Z S) ConvBertLayercsnt|j|_d|_t||_|j|_|j|_|jrV|jsLt|dt||_ t ||_ t ||_ dS)Nrz> should be used as a decoder model if cross attention is added)rHrIchunk_size_feed_forward seq_len_dimr attention is_decoderadd_cross_attention TypeErrorcrossattentionr intermediaterrrYr[r>r?rI#s    zConvBertLayer.__init__NF)rrrrencoder_attention_maskrr_c Cs|j||||d}|d}|dd} |jrx|dk rxt|dsNtd|d||||||} | d}| | dd} t|j|j|j|} | f| } | S)Nrrrrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`) rrrbAttributeErrorrrfeed_forward_chunkrr) rZrrrrrrZself_attention_outputsrrZcross_attention_outputs layer_outputr>r>r?rh1s<     zConvBertLayer.forwardcCs||}|||}|Sr|)rr)rZrZintermediate_outputrr>r>r?rYs  z ConvBertLayer.feed_forward_chunk)NNNNF) rirjrkrIr-rrrnrrrhrror>r>r[r?r"s   (rc sjeZdZfddZd ejeejeejeejeejeeeeeee e e fd ddZ Z S) r}cs:t|_tfddtjD|_d|_dS)Ncsg|] }tqSr>)r).0_r4r>r? csz,ConvBertEncoder.__init__..F) rHrIr4rZ ModuleListr+r,layerr~rYr[rr?rI`s  zConvBertEncoder.__init__NFT) rrrrrroutput_hidden_states return_dictr_c  s|rdnd} rdnd} r(|jjr(dnd} t|jD]\} } |rL| |f} |dk r\|| nd}|jr|jrfdd}tjj|| |||||}n| |||||}|d}r6| |df} |jjr6| |df} q6|r| |f} |st dd|| | | fDSt || | | d S) Nr>csfdd}|S)Ncs|fSr|r>)inputs)rzrr>r?custom_forward}szNConvBertEncoder.forward..create_custom_forward..custom_forwardr>)rzrr)rzr?create_custom_forward|sz6ConvBertEncoder.forward..create_custom_forwardrrr css|]}|dk r|VqdSr|r>)rvr>r>r? sz*ConvBertEncoder.forward..)Zlast_hidden_stater attentionsZcross_attentions) r4r enumeraterr~Ztrainingr-utils checkpointtupler )rZrrrrrrrrZall_hidden_statesZall_self_attentionsZall_cross_attentionsiZ layer_moduleZlayer_head_maskrZ layer_outputsr>rr?rhfsT        zConvBertEncoder.forward)NNNNFFT)rirjrkrIr-rrrnrrrr rhror>r>r[r?r}_s&  r}cs0eZdZfddZejejdddZZS)ConvBertPredictionHeadTransformcsVtt|j|j|_t|jtr6t |j|_ n|j|_ tj |j|j d|_ dSr) rHrIrrurrrtrrr transform_act_fnrPrQrYr[r>r?rIs   z(ConvBertPredictionHeadTransform.__init__rcCs"||}||}||}|Sr|)rrrPrr>r>r?rhs   z'ConvBertPredictionHeadTransform.forwardrr>r>r[r?rs raK This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`ConvBertConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a8 Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zbThe bare ConvBERT Model transformer outputting raw hidden-states without any specific head on top.cseZdZfddZddZddZddZee d e e e e d deejeejeejeejeejeejeeeeeeeee fd d dZZS) ConvBertModelcsPt|t||_|j|jkr4t|j|j|_t ||_ ||_ | dSr|) rHrIrArgrLrrruembeddings_projectr}encoderr4 post_initrYr[r>r?rIs    zConvBertModel.__init__cCs|jjSr|rgrMrZr>r>r?get_input_embeddings sz"ConvBertModel.get_input_embeddingscCs ||j_dSr|r)rZr=r>r>r?set_input_embeddingssz"ConvBertModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)rZZheads_to_prunerrr>r>r? _prune_headsszConvBertModel._prune_headsbatch_size, sequence_lengthr output_typerN) r]rrFrDrr^rrrr_c Csz|dk r |n|jj}|dk r |n|jj}| dk r4| n|jj} |dk rV|dk rVtdn@|dk rt||||} n"|dk r|dd} ntd| \} } |dk r|jn|j} |dkrtj | | d}|dkrt |j dr |j j ddd| f}| | | }|}ntj| tj| d}||| }|||jj}|j ||||d}t |dr`||}|j|||||| d }|S) NzDYou cannot specify both input_ids and inputs_embeds at the same timer!z5You have to specify either input_ids or inputs_embeds)rarFr`)r]rDrFr^r)rrrrr)r4rruse_return_dictrZ%warn_if_padding_and_no_attention_maskrWrar-ZonesrbrgrFrUrVrXZget_extended_attention_maskZ get_head_maskr,rr)rZr]rrFrDrr^rrrrcrrdrarerfZextended_attention_maskrr>r>r?rhsR         zConvBertModel.forward) NNNNNNNNN)rirjrkrIrrrrCONVBERT_INPUTS_DOCSTRINGformatr_CHECKPOINT_FOR_DOCr _CONFIG_FOR_DOCrr-rmrnrrrrhror>r>r[r?rs>   rcs4eZdZdZfddZejejdddZZS)ConvBertGeneratorPredictionszAPrediction module for the generator, made up of two dense layers.cs4ttj|j|jd|_t|j|j|_dSr) rHrIrrPrLrQrurrrYr[r>r?rIbs z%ConvBertGeneratorPredictions.__init__)generator_hidden_statesr_cCs$||}td|}||}|S)NZgelu)rr rP)rZrrr>r>r?rhhs   z$ConvBertGeneratorPredictions.forward) rirjrkrlrIr-rnrhror>r>r[r?r_s rz6ConvBERT Model with a `language modeling` head on top.cseZdZdgZfddZddZddZee de e e e d deejeejeejeejeejeejeejeeeeeeeee fd d d ZZS)ConvBertForMaskedLMzgenerator.lm_head.weightcs>t|t||_t||_t|j|j |_ | dSr|) rHrIrrqrgenerator_predictionsrrurLrKgenerator_lm_headrrYr[r>r?rIts    zConvBertForMaskedLM.__init__cCs|jSr|rrr>r>r?get_output_embeddings~sz)ConvBertForMaskedLM.get_output_embeddingscCs ||_dSr|r )rZrMr>r>r?set_output_embeddingssz)ConvBertForMaskedLM.set_output_embeddingsrrN r]rrFrDrr^labelsrrrr_c  Cs| dk r | n|jj} ||||||||| | } | d} || } || } d}|dk rzt}|| d|jj|d}| s| f| dd}|dk r|f|S|St || | j | j dS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Nrr!rlosslogitsrr) r4rrqrrrrrrKrrr)rZr]rrFrDrr^r rrrrZgenerator_sequence_outputZprediction_scoresrloss_fctrr>r>r?rhs8   zConvBertForMaskedLM.forward) NNNNNNNNNN)rirjrkZ_tied_weights_keysrIr r rrrrrrrrr-rmrnrrrrhror>r>r[r?rpsB   rcs4eZdZdZfddZejejdddZZS)ConvBertClassificationHeadz-Head for sentence-level classification tasks.csZtt|j|j|_|jdk r,|jn|j}t||_ t|j|j |_ ||_ dSr|) rHrIrrurrclassifier_dropoutrSrRrT num_labelsout_projr4rZr4rr[r>r?rIs  z#ConvBertClassificationHead.__init__rcKsR|dddddf}||}||}t|jj|}||}||}|S)Nr)rTrr r4rr)rZrrrr>r>r?rhs    z"ConvBertClassificationHead.forwardrr>r>r[r?rs rz ConvBERT Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cseZdZfddZeedeee e dd e e j e e je e j e e j e e je e je e j e ee ee eeee fd ddZZS) !ConvBertForSequenceClassificationcs:t||j|_||_t||_t||_|dSr|) rHrIrr4rrqr classifierrrYr[r>r?rIs    z*ConvBertForSequenceClassification.__init__rrNr c  Csr| dk r | n|jj} |j|||||||| | d } | d} || } d}|dk r.|jjdkr|jdkrpd|j_n4|jdkr|jtjks|jtj krd|j_nd|j_|jjdkrt }|jdkr|| | }n || |}nN|jjdkrt }|| d|j| d}n|jjdkr.t}|| |}| s^| f| dd}|dk rZ|f|S|St|| | j| jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NrrFrDrr^rrrrrZ regressionZsingle_label_classificationZmulti_label_classificationr!r)r4rrqrZ problem_typerrGr-rXrr squeezerrrrrrrZr]rrFrDrr^r rrrrsequence_outputrrrrr>r>r?rhsT      "     z)ConvBertForSequenceClassification.forward) NNNNNNNNNN)rirjrkrIrrrrrrrrr-rmrnrrrrhror>r>r[r?rs<  rz ConvBERT Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks. cseZdZfddZeedeee e dd e e j e e je e j e e j e e je e je e j e ee ee eeee fd ddZZS) ConvBertForMultipleChoicecs<t|t||_t||_t|jd|_ | dS)Nr) rHrIrrqrsequence_summaryrrurrrrYr[r>r?rI@s    z"ConvBertForMultipleChoice.__init__z(batch_size, num_choices, sequence_lengthrNr c  Cst| dk r | n|jj} |dk r&|jdn|jd} |dk rJ|d|dnd}|dk rh|d|dnd}|dk r|d|dnd}|dk r|d|dnd}|dk r|d|d|dnd}|j|||||||| | d } | d} || }||}|d| }d}|dk r0t}|||}| s`|f| dd}|dk r\|f|S|St ||| j | j dS)aJ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nrr!rrrr) r4rr7rrWrqrrrrrr)rZr]rrFrDrr^r rrrZ num_choicesrrZ pooled_outputrZreshaped_logitsrrrr>r>r?rhJsL      z!ConvBertForMultipleChoice.forward) NNNNNNNNNN)rirjrkrIrrrrrrrrr-rmrnrrrrhror>r>r[r?r8s@  rz ConvBERT Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. cseZdZfddZeedeee e dd e e j e e je e j e e j e e je e je e j e ee ee eeee fd ddZZS) ConvBertForTokenClassificationcs^t||j|_t||_|jdk r.|jn|j}t||_ t |j |j|_ | dSr|)rHrIrrrqrrSrrRrTrurrrrr[r>r?rIs   z'ConvBertForTokenClassification.__init__rrNr c  Cs| dk r | n|jj} |j|||||||| | d } | d} || } || } d}|dk rxt}|| d|j|d}| s| f| dd}|dk r|f|S|St|| | j | j dS)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrrr!rr) r4rrqrTrrrrrrrrr>r>r?rhs8   z&ConvBertForTokenClassification.forward) NNNNNNNNNN)rirjrkrIrrrrrrrrr-rmrnrrrrhror>r>r[r?rs<   rz ConvBERT Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). cseZdZfddZeedeee e dd e e j e e je e j e e j e e je e je e j e e j e ee ee eeee fd ddZZS) ConvBertForQuestionAnsweringcs<t||j|_t||_t|j|j|_| dSr|) rHrIrrrqrrur qa_outputsrrYr[r>r?rIs   z%ConvBertForQuestionAnswering.__init__rrN) r]rrFrDrr^start_positions end_positionsrrrr_c  CsP| dk r | n|jj} |j||||||| | | d } | d} || }|jddd\}}|d}|d}d}|dk r|dk rt|dkr|d}t|dkr|d}|d}| d|}| d|}t |d}|||}|||}||d}| s:||f| dd}|dk r6|f|S|St |||| j | j d S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Nrrrr!r)Z ignore_indexr )r start_logits end_logitsrr)r4rrqr!splitrrrrWclamprrrr)rZr]rrFrDrr^r"r#rrrrrrr$r%Z total_lossZ ignored_indexrZ start_lossZend_lossrr>r>r?rhsP           z$ConvBertForQuestionAnswering.forward) NNNNNNNNNNN)rirjrkrIrrrrrrrrr-rmrnrrrrhror>r>r[r?r s@  r )Frlrr%operatorrtypingrrrr-Ztorch.utils.checkpointrZtorch.nnrrr Z activationsr r Zmodeling_outputsr rrrrrZmodeling_utilsrrZ pytorch_utilsrrrrrrrrZconfiguration_convbertrZ get_loggerrir#rrZ&CONVBERT_PRETRAINED_MODEL_ARCHIVE_LISTr@ModulerArprrrrrrrrr}rZCONVBERT_START_DOCSTRINGrrrrrrrrr r>r>r>r?s     |< -=K 5`PXVJ