U 9%ek @sdZddlZddlmZddlmZmZmZmZddl Z ddl Z ddl m Z ddl mZmZmZddlmZdd lmZmZmZmZmZmZdd lmZdd lmZmZmZm Z m!Z!m"Z"d d l#m$Z$e!%e&Z'dZ(dZ)ddddddddddg Z*dZ+ddZ,Gddde j-Z.Gddde j-Z/e j0e1e1e j0d d!d"Z2Gd#d$d$e j-Z3Gd%d&d&e j-Z4Gd'd(d(e j-Z5Gd)d*d*e j-Z6dTe j0e1e1e7e7e j0d-d.d/Z8Gd0d1d1e j-Z9Gd2d3d3e j-Z:Gd4d5d5eZ;Gd6d7d7e j-Zd;Z?edGd=d>d>e;Z@ed?e>Gd@dAdAe;ZAedBe>GdCdDdDe;ZBedEe>GdFdGdGe;ZCedHe>GdIdJdJe;ZDedKe>GdLdMdMe;ZEedNe>GdOdPdPe;ZFedQe>GdRdSdSe;ZGdS)Uz" PyTorch Funnel Transformer model.N) dataclass)ListOptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BaseModelOutputMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) ModelOutputadd_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) FunnelConfigrzfunnel-transformer/smallfunnel-transformer/small-basezfunnel-transformer/mediumzfunnel-transformer/medium-basezfunnel-transformer/intermediatez$funnel-transformer/intermediate-basezfunnel-transformer/largezfunnel-transformer/large-basezfunnel-transformer/xlarge-basezfunnel-transformer/xlargeg.AcCszddl}ddl}ddl}Wn tk r<tdYnXtj|}t d||j |}g}g} |D]@\} } t d| d| |j || } | | | | qrddd d d d d dddddddd} t|| D]\} } | d} tdd| Dr t dd| q| ddkr0q|}d}| ddD] }t|ts|d|rt|d|d}||jkrd}||j|kr||j|8}|d7}q|j||}n||j8}|j|}n|dkrt|tr|j}qhnd|| krt|| |}nJzt||}Wn:tk rbt dd| | j!d}YqhYnXqD|st"|j!t"| j!kr| #|j!} |dkr|$| } t%&| |_'q|S) z'Load tf checkpoints in a pytorch model.rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z&Converting TensorFlow checkpoint from zLoading TF weight z with shape k_headq_headv_head post_projlinear_1linear_2 attentionffnweightbiasword_embeddings embeddings)kqvoZlayer_1Zlayer_2Zrel_attnffkernelgammabetaZ lookup_tableZword_embeddinginput/css|]}|dkVqdS))Zadam_vZadam_mZAdamWeightDecayOptimizerZAdamWeightDecayOptimizer_1Z global_stepN).0nr3r3i/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/transformers/models/funnel/modeling_funnel.py rsz,load_tf_weights_in_funnel..z Skipping generatorFrz layer_\d+z layer_(\d+)rTr.)(renumpyZ tensorflow ImportErrorloggererrorospathabspathinfotrainZlist_variablesZ load_variableappendzipsplitanyjoin isinstanceFunnelPositionwiseFFN fullmatchintsearchgroupsZnum_hidden_layers block_sizesblockslayersFunnelRelMultiheadAttentionr_kernelgetattrAttributeErrorprintshapelenreshapeZ transposetorchZ from_numpydata)modelconfigZtf_checkpoint_pathr:nptfZtf_pathZ init_varsnamesZarraysnamerWarrayZ _layer_mapZpointerZskippedZm_name layer_indexZ block_idxr3r3r6load_tf_weights_in_funnelDs                rdcsFeZdZeddfdd ZdeejeejejdddZZ S) FunnelEmbeddingsNr]returncsHttj|j|j|jd|_tj|j |j d|_ t |j |_dS)N) padding_idxZeps)super__init__r Embedding vocab_size hidden_sizeZ pad_token_idr' LayerNormd_modellayer_norm_eps layer_normDropouthidden_dropoutdropoutselfr] __class__r3r6rks zFunnelEmbeddings.__init__) input_ids inputs_embedsrgcCs*|dkr||}||}||}|SN)r'rrru)rwrzr{r(r3r3r6forwards    zFunnelEmbeddings.forward)NN) __name__ __module__ __qualname__rrkrrZTensorr} __classcell__r3r3rxr6resrecseZdZUdZdZeed<eddfdd Zd$e j e e j e e j e e j dd d Z e j e j d d d Zee je jee e j eee j fdddZe j edddZd%e j eee j dddZee j e e j ee j feee eeefe j dddZd&ee j e e j ee j feee j dddZe e j e e j e e j fdd d!Ze e j e e j dd"d#ZZS)'FunnelAttentionStructurez> Contains helpers for `FunnelRelMultiheadAttention `. cls_token_type_idNrfcs6t||_t|j|_t|j|_d|_dSr|) rjrkr]rrsrt sin_dropout cos_dropout pooling_multrvrxr3r6rks  z!FunnelAttentionStructure.__init__)r{attention_masktoken_type_idsrgcCsvd|_|d|_}|||j|j}|dk r:||nd}|jjrft j | |d|dgdnd}||||fS)zCReturns the attention inputs associated to the inputs of the model.rN)rrrr) rsizeseq_lenget_position_embedsdtypedevicetoken_type_ids_to_matr] separate_clsr functionalpadZnew_ones)rwr{rrrposition_embedstoken_type_matcls_maskr3r3r6init_attention_inputss "z.FunnelAttentionStructure.init_attention_inputs)rrgcCs^|dddddf|dddfk}||jk}|dddddf|dddfB}||BS)z-Convert `token_type_ids` to `token_type_mat`.N)r)rwrrZcls_idsZcls_matr3r3r6rs& &z.FunnelAttentionStructure.token_type_ids_to_mat)rrrrgcCs|jj}|jjdkrtjd|d||d}tjd|dd||d}dd||d}|dddf|d}t|} || } t|} || } tj | | gd d } tj | | gd d }tj | | gd d }tj | | gd d }| |||fStjd|dd||d}dd||d}tj| d|dd||d}|d}|dddf|d}|t|} |t|} tj | | gd d }tjd|||d}|}g}t d|jj D]}|dkrd}n^| ||}d|d}|j |||dd }|dddf|}||d|}t|d|}|}d|}| ||}|dddf|}||d|}t|d|}|||gq|SdS) a Create and cache inputs related to relative position encoding. Those are very different depending on whether we are using the factorized or the relative shift attention: For the factorized attention, it returns the matrices (phi, pi, psi, omega) used in the paper, appendix A.2.2, final formula. For the relative shift attention, it returns all possible vectors R used in the paper, appendix A.2.1, final formula. Paper link: https://arxiv.org/abs/2006.03236 factorizedr?rrrri'Ndim)shift)r]rpattention_typerZarangesinrcosrcatrangeZ num_blocksstride_pool_pos relative_posexpandrgatherrD)rwrrrrpZpos_seqZfreq_seqZinv_freqZsinusoidZ sin_embedZ sin_embed_dZ cos_embedZ cos_embed_dphipsipiomegaZ rel_pos_idZ zero_offsetZ pos_embedpos pooled_posZposition_embeds_list block_indexZposition_embeds_poolingstrideZrel_posZposition_embeds_no_poolingr3r3r6rsV          z,FunnelAttentionStructure.get_position_embeds)pos_idrcCsj|jjrX|d| dg}|jjr2|ddn |dd}t||dddgdS|dddSdS)ze Pool `pos_id` while keeping the cls token separate (if `config.separate_cls=True`). rrrNr)r]rZ new_tensor truncate_seqrZr)rwrrZcls_posZ pooled_pos_idr3r3r6r+s  z(FunnelAttentionStructure.stride_pool_posr)rrrrgc Csb|dkr |}|d|d}|t|}|||}|d|d}tj||d| tj|jdS)zV Build the relative positional vector between `pos` and `pooled_pos`. Nrrrr)rXrZrlongr) rwrrrrZ ref_pointZ num_removeZmax_distZmin_distr3r3r6r:s  z%FunnelAttentionStructure.relative_pos)tensoraxisrgcs|dkr dStttfr4D]}||}q|St|ttfr^t|fdd|DS|j;jjrjjrt dddn t ddd}t dg|g}jjrt dgt ddg}t j |||gd}||S)zT Perform pooling by stride slicing the tensor along the given axis. Nc3s|]}|VqdSr|) stride_poolr4xrrwr3r6r7[sz7FunnelAttentionStructure.stride_pool..rrr)r) rIlisttuplertypendimr]rrslicerZr)rwrrZaxZ axis_sliceZ enc_sliceZ cls_slicer3rr6rHs  &z$FunnelAttentionStructure.stride_poolmean)rmoderrgcsdkr dStttfr:tfddDSjjrjjr^ddddfn}tjddddf|gddj }|dkrddddddfn$|dkrЈdddddddfdfd krt j j d d nLd krt j j d d n,d kr:t j j  d d  ntd|dkrdddddddfS|dkr~dddfSS)z3Apply 1D pooling to a tensor of size [B x T (x H)].Nc3s|]}jdVqdS))rrN) pool_tensorrrrwrrr3r6r7rsz7FunnelAttentionStructure.pool_tensor..rrrrr rT)rZ ceil_modemaxminz0The supported modes are 'mean', 'max' and 'min'.r)rIrrrr]rrrZrrrrZ avg_pool2dZ max_pool2dNotImplementedError)rwrrrsuffixrr3rr6ris2  "    z$FunnelAttentionStructure.pool_tensor)attention_inputsrgcCs|\}}}}|jjrl|jjdkr@||ddd|dd}||d}||d}|j||jjd}nf|jd9_|jjdkr||d}||ddg}||ddg}|j|dd}|j||jjd}||||f}||fS)zTPool `output` and the proper parts of `attention_inputs` before the attention layer.rNrrrrr)r] pool_q_onlyrrrZ pooling_typer)rwoutputrrrrrr3r3r6pre_attention_poolings         z.FunnelAttentionStructure.pre_attention_poolingcCs|\}}}}|jjrt|jd9_|jjdkrN|dd||ddd}||d}||d}|j|dd}||||f}|S)zFPool the proper parts of `attention_inputs` after the attention layer.rrNrrrr)r]rrrrr)rwrrrrrr3r3r6post_attention_poolings      z/FunnelAttentionStructure.post_attention_pooling)NN)Nr)rr)r~rr__doc__rrL__annotations__rrkrZrrrrrrrrrrrrrstrrrrrr3r3rxr6rsF      P " ' r)positional_attn context_lenrrgcCsn|j\}}}}t|||||g}|dddd|dddf}t||||||g}|dd|f}|S)N.)rWrZrY)rrr batch_sizen_headrZ max_rel_lenr3r3r6_relative_shift_gathers  rc sjeZdZeeddfdd ZdddZdddZdej ej ej e ej e e ej d fd d d Z Z S)rRNr]rrgcs*t||_||_|j|j|j}}}t|j |_ t|j |_ tj |||dd|_ t ||||_ t ||||_tt||g|_tt||g|_tt|||g|_tt||g|_ttd||g|_t ||||_tj||jd|_d|d|_dS)NF)r&rrirg?)rjrkr]rrprd_headrrsrtattention_dropoutLinearrrr ParameterrZzerosr_w_biasr_r_biasrSr_s_bias seg_embedr rorqrrscale)rwr]rrprrrxr3r6rks" z$FunnelRelMultiheadAttention.__init__cCs|jjdkr|\}}}}|j|j} |j} td|| | } | |dddf} | |dddf} td| |td| |}nf|jd|krdnd}||j|d}|j|j}|j} td|| }td|||}t |||}|dk r||9}|S) z5Relative attention score for the positional encodingsrzbinh,dnh->bindNz bind,jd->bnijrrz td,dnh->tnhzbinh,tnh->bnit) r]rrrrSrZeinsumrWrr)rwrrrrrrrruZw_rZ q_r_attentionZq_r_attention_1Zq_r_attention_2rrr9r+Zr_headr3r3r6relative_positional_attentions,     z9FunnelRelMultiheadAttention.relative_positional_attentionc Cs|dkr dS|j\}}}|j|j}td|||j}|dddf||jd||g}tj|ddd\} } t|| |j| |j} |dk r| |9} | S)z/Relative attention score for the token_type_idsNrzbind,snd->bnisrrrr) rWrrrZrrrrFwhere) rwrrrrrrrZtoken_type_biasZdiff_token_typeZsame_token_typetoken_type_attnr3r3r6relative_token_type_attentions  $ z9FunnelRelMultiheadAttention.relative_token_type_attentionF.)querykeyvalueroutput_attentionsrgcCsn|\}}}} |j\} } } |jd} |jj|jj}}||| | ||}||| | ||}||| | ||}||j}|j |j}t d|||}| ||| | }| ||| }|||}|j}|}|dk r|td|ddddf}t j|d|d}||}t d||}||| | ||}||}|||}|rh||fS|fS)Nrzbind,bjnd->bnijr)rrzbnij,bjnd->bind)rWr]rrrviewrrrrrZrrrrfloatINFZsoftmaxrr rYrtrr)rwrrrrrrrrrrr_rrrrrrrZ content_scorerrZ attn_scorerZ attn_probZattn_vecZattn_outrr3r3r6r}s0       "  z#FunnelRelMultiheadAttention.forward)N)N)F)r~rrrrLrkrrrZrrboolr}rr3r3rxr6rRs *  rRcs8eZdZeddfdd ZejejdddZZS)rJNrfcsltt|j|j|_t|j|_ t |j |_ t|j|j|_ t |j |_t|j|j|_dSr|)rjrkrrrpZd_innerr!r hidden_actactivation_functionrsactivation_dropoutr"rtrurorqrrrvrxr3r6rkTs  zFunnelPositionwiseFFN.__init__hiddenrgcCs@||}||}||}||}||}|||Sr|)r!rrr"rurr)rwrhr3r3r6r}]s      zFunnelPositionwiseFFN.forward) r~rrrrkrZrr}rr3r3rxr6rJSs rJcsDeZdZeeddfdd Zd ejejejee dddZ Z S) FunnelLayerNrcs$tt|||_t||_dSr|)rjrkrRr#rJr$)rwr]rrxr3r6rkgs  zFunnelLayer.__init__F)rrrrrgcCs8|j|||||d}||d}|r2||dfS|fS)Nrrr)r#r$)rwrrrrrZattnrr3r3r6r}lszFunnelLayer.forward)F) r~rrrrLrkrZrrrr}rr3r3rxr6rfs rc sVeZdZeddfdd Zd ejeejeejeeee e e fddd Z Z S) FunnelEncoderNrfcs>t|_t|_tfddtjD|_ dS)Ncs.g|]&\}tfddt|DqS)csg|]}tqSr3rr4r)rr]r3r6 sz5FunnelEncoder.__init__...)r ModuleListr)r4 block_sizer])rr6rsz*FunnelEncoder.__init__..) rjrkr]rattention_structurerr enumeraterOrPrvrxrr6rkzs   zFunnelEncoder.__init__FT)r{rrroutput_hidden_states return_dictrgc Csl||}|jj|||d}|}|r*|fnd} |r6dnd} t|jD]\} } |d|jjr`dndk} | op| dk} | r|j||\}}t| D]\}}t |jj | D]}|dko|dko| }|r|}|jj r|n|}}n |}}}||||||d}|d}|r|j |}|r,| |dd} |r| |f} qqqD|s^t dd|| | fDSt|| | d S) Nrrr3rrrrcss|]}|dk r|VqdSr|r3r4r+r3r3r6r7sz(FunnelEncoder.forward..last_hidden_state hidden_states attentions)Ztype_asrrrrPrr]rrrZ block_repeatsrrrr )rwr{rrrrrrrall_hidden_statesall_attentionsrblockZ pooling_flagZ pooled_hiddenrclayerZ repeat_indexZ do_poolingrrr layer_outputr3r3r6r}sF     zFunnelEncoder.forward)NNFFTr~rrrrkrZrrrrrr r}rr3r3rxr6rys rTF)rr target_lenrrrgc Cs|dkr |S|r8|ddddf}|ddddf}tj||dd}|r|rntj|ddd|dddf}|ddd|df}tj||gdd}n|ddd|f}|S)z{ Upsample tensor `x` to match `target_len` by repeating the tokens `stride` time on the sequence length dimension. rN)Zrepeatsrrr)rZZrepeat_interleaverrrr)rrrrrclsrr3r3r6upsamplesrc sZeZdZeddfdd Zd ejejeejeejeeee e e fddd Z Z S) FunnelDecoderNrfcs>t|_t|_tfddtjD|_ dS)Ncsg|]}tdqS)rrrrr3r6rsz*FunnelDecoder.__init__..) rjrkr]rrrrrZnum_decoder_layersrQrvrxrr6rks  zFunnelDecoder.__init__FT) final_hiddenfirst_block_hiddenrrrrrrgcCst|dt|jjd|jd|jj|jjd}||} |rB| fnd} |rNdnd} |jj| ||d} |j D]@} | | | | | |d}|d} |r| |dd} |rj| | f} qj|st dd | | | fDSt | | | d S) Nrr)rrrrr3rrrcss|]}|dk r|VqdSr|r3rr3r3r6r7sz(FunnelDecoder.forward..r) rrXr]rOrWrrrrrQrr )rwrrrrrrrZupsampled_hiddenrrr rr r r3r3r6r}s2    zFunnelDecoder.forward)NNFFTr r3r3rxr6rs   rcs<eZdZdZeddfdd ZejejdddZZ S) FunnelDiscriminatorPredictionszEPrediction module for the discriminator, made up of two dense layers.Nrfcs6t||_t|j|j|_t|jd|_dSNr)rjrkr]rrrpdensedense_predictionrvrxr3r6rks z'FunnelDiscriminatorPredictions.__init__)discriminator_hidden_statesrgcCs,||}t|jj|}||}|Sr|)rr r]rrsqueeze)rwrrlogitsr3r3r6r}s z&FunnelDiscriminatorPredictions.forward) r~rrrrrkrZrr}rr3r3rxr6rsrc@s$eZdZdZeZeZdZddZ dS)FunnelPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. funnelcCsj|jj}|ddkrt|dddk rp|jjdkrV|jj\}}t dt ||}n|jj}t j j |j|dt|dddk rt j |jdn|dkr t j j|j|jjd t j j|j|jjd t j j|j|jjd t j j|j|jjd t j j|j|jjd nZ|d krf|jjdkr(dn|jj}t j j |jj|d|jjdk rf|jjj|jdS) Nrrr%r)stdr&grR)bre)ryr~findrTr]Zinitializer_stdr%rWr^sqrtrrinitZnormal_Z constant_r&Zuniform_rZinitializer_rangerrSrrr'rhr[Zzero_)rwmodule classnameZfan_outZfan_inrr3r3r6 _init_weightss*    z#FunnelPreTrainedModel._init_weightsN) r~rrrr config_classrdZload_tf_weightsZbase_model_prefixr$r3r3r3r6rs rcs:eZdZeeddfdd ZejejdddZZ S)FunnelClassificationHeadN)r]n_labelsrgcs>tt|j|j|_t|j|_t|j||_ dSr|) rjrkrrrp linear_hiddenrsrtru linear_out)rwr]r'rxr3r6rk3s z!FunnelClassificationHead.__init__rcCs(||}t|}||}||Sr|)r(rZtanhrur))rwrr3r3r6r}9s   z FunnelClassificationHead.forward) r~rrrrLrkrZrr}rr3r3rxr6r&2sr&c@s^eZdZUdZdZeejed<dZ ejed<dZ ee ejed<dZ ee ejed<dS)FunnelForPreTrainingOutputa Output type of [`FunnelForPreTraining`]. Args: loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss of the ELECTRA-style objective. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Prediction scores of the head (scores for each token before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlossrrr) r~rrrr,rrZZ FloatTensorrrrrrr3r3r3r6r+@s r+a( The Funnel Transformer model was proposed in [Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing](https://arxiv.org/abs/2006.03236) by Zihang Dai, Guokun Lai, Yiming Yang, Quoc V. Le. This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`FunnelConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z The base Funnel Transformer Model transformer outputting raw hidden-states without upsampling head (also called decoder) or any task-specific head on top. cseZdZeddfdd ZejdddZejddd d Ze e d e d e ed deejeejeejeejeejeejeeeeeeeee fd ddZZS)FunnelBaseModelNrfcs,t|t||_t||_|dSr|)rjrkrer(rencoder post_initrvrxr3r6rks   zFunnelBaseModel.__init__rgcCs|jjSr|r(r'rwr3r3r6get_input_embeddingssz$FunnelBaseModel.get_input_embeddingsnew_embeddingsrgcCs ||j_dSr|r1rwr5r3r3r6set_input_embeddingssz$FunnelBaseModel.set_input_embeddingsbatch_size, sequence_lengthr checkpoint output_typer%) rzrr position_ids head_maskr{rrrrgc Cs|dk r |n|jj}|dk r |n|jj}| dk r4| n|jj} |dk rV|dk rVtdn@|dk rt||||} n"|dk r|dd} ntd|dk r|jn|j} |dkrtj | | d}|dkrtj | tj | d}|dkr| |}|j |||||| d} | S)NDYou cannot specify both input_ids and inputs_embeds at the same timer5You have to specify either input_ids or inputs_embedsrrrrrrr)r]rruse_return_dict ValueError%warn_if_padding_and_no_attention_maskrrrZonesrrr(r.) rwrzrrr<r=r{rrr input_shaperencoder_outputsr3r3r6r}s8     zFunnelBaseModel.forward) NNNNNNNNN)r~rrrrkrrlr3r7rFUNNEL_INPUTS_DOCSTRINGformatrr _CONFIG_FOR_DOCrrZrrrrr}rr3r3rxr6r-s<   r-zlThe bare Funnel Transformer Model transformer outputting raw hidden-states without any specific head on top.c seZdZeddfdd ZejdddZejddd d Ze e d e e eed deejeejeejeejeeeeeeeeefd ddZZS) FunnelModelNrfcs<t|||_t||_t||_t||_| dSr|) rjrkr]rer(rr.rdecoderr/rvrxr3r6rks     zFunnelModel.__init__r0cCs|jjSr|r1r2r3r3r6r3sz FunnelModel.get_input_embeddingsr4cCs ||j_dSr|r1r6r3r3r6r7sz FunnelModel.set_input_embeddingsr8r9)rzrrr{rrrrgc Cs|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|dk rV|dk rVtdn@|dk rt||||}n"|dk r|dd}ntd|dk r|jn|j} |dkrtj || d}|dkrtj |tj | d}|dkr| |}|j ||||d|d} |j| d| d |jjd|||||d } |sd} | df} |rf| d 7} | | d | | f} |r| d 7} | | d | | f} | St| d|r| j| jnd|r| j| jndd S) Nr>rr?r@rTrArr)rrrrrrrrr)r]rrrBrCrDrrrZrErrr(r.rLrOr rr)rwrzrrr{rrrrFrrGZdecoder_outputsidxoutputsr3r3r6r}sj       zFunnelModel.forward)NNNNNNN)r~rrrrkrrlr3r7rrHrIr_CHECKPOINT_FOR_DOCr rJrrZrrrrr}rr3r3rxr6rKs4   rKz Funnel Transformer model with a binary classification head on top as used during pretraining for identifying generated tokens. cseZdZeddfdd Zeedee e dd e e j e e j e e j e e j e e j e ee ee eeee fd dd ZZS) FunnelForPreTrainingNrfcs,t|t||_t||_|dSr|)rjrkrKrrdiscriminator_predictionsr/rvrxr3r6rkXs   zFunnelForPreTraining.__init__r8)r;r% rzrrr{labelsrrrrgc  Cs|dk r |n|jj}|j|||||||d} | d} || } d} |dk rt} |dk r|d| jddk}| d| jd|}||}| ||} n| | d| jd|} |s| f| dd}| dk r| f|S|St | | | j | j dS)a4 labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the ELECTRA-style loss. Input should be a sequence of tokens (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates the token is an original token, - 1 indicates the token was replaced. Returns: Examples: ```python >>> from transformers import AutoTokenizer, FunnelForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("funnel-transformer/small") >>> model = FunnelForPreTraining.from_pretrained("funnel-transformer/small") >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> logits = model(**inputs).logits ```Nrrr{rrrrrrr,rrr) r]rBrrQrrrrWrr+rr)rwrzrrr{rSrrrrZdiscriminator_sequence_outputrr,loss_fctZ active_lossZ active_logitsZ active_labelsrr3r3r6r}`s<#  zFunnelForPreTraining.forward)NNNNNNNN)r~rrrrkrrHrIrr+rJrrZrrrrr}rr3r3rxr6rPWs,   rPz@Funnel Transformer Model with a `language modeling` head on top.cseZdZdgZeddfdd ZejdddZej dd d d Z e e d eeeed ddeejeejeejeejeejeeeeeeeeefd ddZZS)FunnelForMaskedLMzlm_head.weightNrfcs4t|t||_t|j|j|_| dSr|) rjrkrKrrrrprmlm_headr/rvrxr3r6rks  zFunnelForMaskedLM.__init__r0cCs|jSr|rXr2r3r3r6get_output_embeddingssz'FunnelForMaskedLM.get_output_embeddingsr4cCs ||_dSr|rYr6r3r3r6set_output_embeddingssz'FunnelForMaskedLM.set_output_embeddingsr8z)r:r;r%maskrRc  Cs|dk r |n|jj}|j|||||||d} | d} || } d} |dk rlt} | | d|jj|d} |s| f| dd}| dk r| f|S|St| | | j| j dS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` NrTrrrrU) r]rBrrXr rrmrrr)rwrzrrr{rSrrrrNrZprediction_logitsZmasked_lm_lossrVrr3r3r6r}s2  zFunnelForMaskedLM.forward)NNNNNNNN)r~rrZ_tied_weights_keysrrkrrrZrlr[rrHrIrrOrrJrrZrrrrr}rr3r3rxr6rWs<   rWz Funnel Transformer Model with a sequence classification/regression head on top (two linear layer on top of the first timestep of the last hidden state) e.g. for GLUE tasks. cseZdZeddfdd Zeedede e dd e e j e e j e e j e e j e e j e ee ee eeee fd d d ZZS) FunnelForSequenceClassificationNrfcs>t||j|_||_t||_t||j|_|dSr|) rjrk num_labelsr]r-rr& classifierr/rvrxr3r6rks   z(FunnelForSequenceClassification.__init__r8rr9rRc  Cs~|dk r |n|jj}|j|||||||d} | d} | dddf} || } d} |dk r:|jjdkr|jdkr|d|j_n4|jdkr|jtjks|jtj krd|j_nd|j_|jjdkrt }|jdkr|| | } n || |} nN|jjdkrt }|| d|j| d} n|jjdkr:t}|| |} |sj| f| dd}| dk rf| f|S|St| | | j| jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NrTrrZ regressionZsingle_label_classificationZmulti_label_classificationrrU)r]rBrr_Z problem_typer^rrZrrLr rr rrrrr)rwrzrrr{rSrrrrNr pooled_outputrr,rVrr3r3r6r}sR      "     z'FunnelForSequenceClassification.forward)NNNNNNNN)r~rrrrkrrHrIrrrJrrZrrrrr}rr3r3rxr6r]s4  r]z Funnel Transformer Model with a multiple choice classification head on top (two linear layer on top of the first timestep of the last hidden state, and a softmax) e.g. for RocStories/SWAG tasks. cseZdZeddfdd Zeedede e dd e e j e e j e e j e e j e e j e ee ee eeee fd d d ZZS) FunnelForMultipleChoiceNrfcs.t|t||_t|d|_|dSr)rjrkr-rr&r_r/rvrxr3r6rkXs   z FunnelForMultipleChoice.__init__z(batch_size, num_choices, sequence_lengthrr9rRc  CsX|dk r |n|jj}|dk r&|jdn|jd} |dk rJ|d|dnd}|dk rh|d|dnd}|dk r|d|dnd}|dk r|d|d|dnd}|j|||||||d} | d} | dddf} || } | d| }d}|dk rt}|||}|sD|f| dd}|dk r@|f|S|St||| j | j dS)aJ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) NrrrTrrU) r]rBrWrrrr_r rrr)rwrzrrr{rSrrrZ num_choicesrNrr`rZreshaped_logitsr,rVrr3r3r6r}`sF     zFunnelForMultipleChoice.forward)NNNNNNNN)r~rrrrkrrHrIrrrJrrZrrrrr}rr3r3rxr6raPs4  raz Funnel Transformer Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. cseZdZeddfdd Zeedee e e dd e e je e je e je e je e je ee ee eeee fd dd ZZS) FunnelForTokenClassificationNrfcsJt||j|_t||_t|j|_t |j |j|_ | dSr|) rjrkr^rKrrrsrtrurrnr_r/rvrxr3r6rks   z%FunnelForTokenClassification.__init__r8r9rRc  Cs|dk r |n|jj}|j|||||||d} | d} || } || } d} |dk rtt} | | d|j|d} |s| f| dd}| dk r| f|S|St| | | j | j dS)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. NrTrrrrU) r]rBrrur_r rr^rrr)rwrzrrr{rSrrrrNrrr,rVrr3r3r6r}s4   z$FunnelForTokenClassification.forward)NNNNNNNN)r~rrrrkrrHrIrrOrrJrrZrrrrr}rr3r3rxr6rcs4  rcz Funnel Transformer Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layer on top of the hidden-states output to compute `span start logits` and `span end logits`). cseZdZeddfdd Zeedee e e dd e e je e je e je e je e je e je ee ee eeee fd dd ZZS) FunnelForQuestionAnsweringNrfcs<t||j|_t||_t|j|j|_| dSr|) rjrkr^rKrrrrn qa_outputsr/rvrxr3r6rks   z#FunnelForQuestionAnswering.__init__r8r9) rzrrr{start_positions end_positionsrrrrgc  CsL| dk r | n|jj} |j||||||| d} | d} || } | jddd\} }| d} |d}d}|dk r|dk rt|dkr| d}t|dkr|d}| d}| d|}| d|}t |d}|| |}|||}||d}| s6| |f| dd}|dk r2|f|S|St || || j | jd S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. NrTrrrr)Z ignore_indexr)r, start_logits end_logitsrr)r]rBrrerFr contiguousrXrZsquezeclampr rrr)rwrzrrr{rfrgrrrrNrrrhriZ total_lossZ ignored_indexrVZ start_lossZend_lossrr3r3r6r}sL           z"FunnelForQuestionAnswering.forward) NNNNNNNNN)r~rrrrkrrHrIrrOrrJrrZrrrrr}rr3r3rxr6rds8  rd)TF)Hrr? dataclassesrtypingrrrrr;r^rZrZtorch.nnrr r Z activationsr Zmodeling_outputsr rrrrrZmodeling_utilsrutilsrrrrrrZconfiguration_funnelrZ get_loggerr~r=rJrOZ$FUNNEL_PRETRAINED_MODEL_ARCHIVE_LISTrrdModulererrrLrrRrJrrrrrrrr&r+ZFUNNEL_START_DOCSTRINGrHr-rKrPrWr]rarcrdr3r3r3r6s         Z@ 1#'Ic RJULB