U 9%e@sdZddlZddlZddlZddlZddlmZmZmZm Z ddl m Z ddl m Z mZddlZddlZddlmZddlmZdd lmZmZdd lmZmZmZmZmZm Z dd l!m"Z"dd l m#Z#erddl$Z$e rddl%Z%e#&e'Z(Gd ddZ)dS)zTensorflow trainer class.N)CallableDictOptionalTuple)ENV_VARS_TRUE_VALUES)is_comet_availableis_wandb_available) PerReplica)TFPreTrainedModel)GradientAccumulatorcreate_optimizer)PREFIX_CHECKPOINT_DIREvalPredictionIntervalStrategyPredictionOutputenable_full_determinismset_seed)TFTrainingArguments)loggingc @seZdZdZd7eeeejj eejj ee e ge feej jeejjjejjjjfdddZejj ddd Zd8eejj ejj d d d Zejj ejj d ddZedddZddZddZd9ejj eeeeeedddZ e ee!fddddZ"d:eejj e ee!fd ddZ#ej$ej$ej$ej$dd d!Z%ej&d"d#Z'ddd$d%Z(d&d'Z)d(d)Z*ej&d*d+Z+e,d,d-Z-e,d.d/Z.d0d1Z/ejj ed d2d3Z0d;eed4d5d6Z1dS)< TFTraineru TFTrainer is a simple but feature-complete training and eval loop for TensorFlow, optimized for 🤗 Transformers. Args: model ([`TFPreTrainedModel`]): The model to train, evaluate or use for predictions. args ([`TFTrainingArguments`]): The arguments to tweak training. train_dataset ([`~tf.data.Dataset`], *optional*): The dataset to use for training. The dataset should yield tuples of `(features, labels)` where `features` is a dict of input features and `labels` is the labels. If `labels` is a tensor, the loss is calculated by the model by calling `model(features, labels=labels)`. If `labels` is a dict, such as when using a QuestionAnswering head model with multiple targets, the loss is instead calculated by calling `model(features, **labels)`. eval_dataset ([`~tf.data.Dataset`], *optional*): The dataset to use for evaluation. The dataset should yield tuples of `(features, labels)` where `features` is a dict of input features and `labels` is the labels. If `labels` is a tensor, the loss is calculated by the model by calling `model(features, labels=labels)`. If `labels` is a dict, such as when using a QuestionAnswering head model with multiple targets, the loss is instead calculated by calling `model(features, **labels)`. compute_metrics (`Callable[[EvalPrediction], Dict]`, *optional*): The function that will be used to compute metrics at evaluation. Must take a [`EvalPrediction`] and return a dictionary string to metric values. tb_writer (`tf.summary.SummaryWriter`, *optional*): Object to write to TensorBoard. optimizers (`Tuple[tf.keras.optimizers.Optimizer, tf.keras.optimizers.schedules.LearningRateSchedule]`, *optional*): A tuple containing the optimizer and the scheduler to use. The optimizer default to an instance of [`tf.keras.optimizers.Adam`] if `args.weight_decay_rate` is 0 else an instance of [`AdamWeightDecay`]. The scheduler will default to an instance of [`tf.keras.optimizers.schedules.PolynomialDecay`] if `args.num_warmup_steps` is 0 else an instance of [`WarmUp`]. NNN)modelargs train_dataset eval_datasetcompute_metrics tb_writer optimizerscCs||_||_||_||_||_|\|_|_t|_d|_ d|_ t j j |_tdt|dk rh||_nt j|jj|_tr|ntddtkrtdtr| ntj!"ddkrtd|jj#rt$|jj%n t&|jj%dS) Nra[The class `TFTrainer` is deprecated and will be removed in version 5 of Transformers. We recommend using native Keras instead, by calling methods like `fit()` and `predict()` directly on the model object. Detailed examples of the Keras style can be found in our examples at https://github.com/huggingface/transformers/tree/main/examples/tensorflowZWANDB_DISABLEDzYou are instantiating a Trainer but W&B is not installed. To use wandb logging, run `pip install wandb && wandb login` see https://docs.wandb.com/huggingface. COMET_MODEZDISABLEDzoTo use comet_ml logging, run `pip/conda install comet_ml` see https://www.comet.ml/docs/python-sdk/huggingface/)'rrrrr optimizer lr_schedulerr gradient_accumulator global_step epoch_loggingtfkerasmetricsSum eval_losswarningswarn FutureWarningrsummaryZcreate_file_writer logging_dirr setup_wandbosgetenvupperrloggerinfor setup_cometenvirongetZfull_determinismrseedr)selfrrrrrrrr;V/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/transformers/trainer_tf.py__init__^s<    zTFTrainer.__init__)returncCs|jdkrtd|jj|jj|_|j|_|jdkrFtd|j j |j|jj dj |j|jj dtjjj}|jj|S)z Returns the training [`~tf.data.Dataset`]. Subclass and override this method if you want to inject some custom behavior. Nz+Trainer: training requires a train_dataset.r6The training dataset must have an asserted cardinality)r9Zdrop_remainder)r ValueErrorrtrain_batch_sizegradient_accumulation_stepstotal_train_batch_size cardinalitynumpynum_train_examplesrepeatshuffler9batchdataloader_drop_lastprefetchr&data experimentalAUTOTUNEstrategyexperimental_distribute_dataset)r:dsr;r;r<get_train_tfdatasets"   zTFTrainer.get_train_tfdataset)rr>cCs|dkr|jdkrtd|dk r&|n|j}|}|dkrHtd|jjrVtjntj}|||jj }| j |jj |jjd t jjj}|jj|||fS)a Returns the evaluation [`~tf.data.Dataset`]. Args: eval_dataset ([`~tf.data.Dataset`], *optional*): If provided, will override *self.eval_dataset*. The dataset should yield tuples of `(features, labels)` where `features` is a dict of input features and `labels` is the labels. If `labels` is a tensor, the loss is calculated by the model by calling `model(features, labels=labels)`. If `labels` is a dict, such as when using a QuestionAnswering head model with multiple targets, the loss is instead calculated by calling `model(features, **labels)`. Subclass and override this method if you want to inject some custom behavior. Nz-Trainer: evaluation requires an eval_dataset.rr?r@)rrArErFrrKmathfloorceileval_batch_sizerHrJrLr&rMrNrOrPrQ)r:r num_examplesapproxstepsrRr;r;r<get_eval_tfdatasets  zTFTrainer.get_eval_tfdataset) test_datasetr>cCs\|}|dkrtdt||jj}||jjt j j j }|jj |||fS)a Returns a test [`~tf.data.Dataset`]. Args: test_dataset ([`~tf.data.Dataset`]): The dataset to use. The dataset should yield tuples of `(features, labels)` where `features` is a dict of input features and `labels` is the labels. If `labels` is a tensor, the loss is calculated by the model by calling `model(features, labels=labels)`. If `labels` is a dict, such as when using a QuestionAnswering head model with multiple targets, the loss is instead calculated by calling `model(features, **labels)`. Subclass and override this method if you want to inject some custom behavior. rr?)rErFrArTrVrrWrJrLr&rMrNrOrPrQ)r:r\rXrZrRr;r;r<get_test_tfdatasets  zTFTrainer.get_test_tfdatasetnum_training_stepsc Csn|jsj|jsj|jjdkr |jjnt||jj}t|jj|||jj |jj |jj |jj |jj d\|_|_dS)a Setup the optimizer and the learning rate scheduler. We provide a reasonable default that works well. If you want to use something else, you can pass a tuple in the TFTrainer's init through `optimizers`, or subclass and override this method. r) adam_beta1 adam_beta2 adam_epsilonZweight_decay_ratepowerN)r!r"r warmup_stepsrTrVZ warmup_ratior learning_rater`rarbZ weight_decayZ poly_power)r:r_rdr;r;r<create_optimizer_and_schedulers  z(TFTrainer.create_optimizer_and_schedulercCs@td|jj|j}tjt dd||jj ddS)ai Setup the optional Weights & Biases (`wandb`) integration. One can subclass and override this method to customize the setup if needed. Find more information `here `__. You can also override the following environment variables: Environment: WANDB_PROJECT: (Optional): str - "huggingface" by default, set this to a custom string to store results in a different project. WANDB_DISABLED: (Optional): boolean - defaults to false, set to "true" to disable wandb entirely. z`Automatic Weights & Biases logging enabled, to disable set os.environ["WANDB_DISABLED"] = "true"Z WANDB_PROJECT huggingface)projectconfignameN) r4r5rrito_dictrZto_sanitized_dictwandbinitr1r2Zrun_name)r:Z combined_dictr;r;r<r0s zTFTrainer.setup_wandbcCstdd}dtddi}d}|dkrDtjf|}tdn.|dkrrtd d |d <tjf|}td |dk r|j|j d d|j |j dd d|j |j j dd ddS)aL Setup the optional Comet.ml integration. Environment: COMET_MODE: (Optional): str - "OFFLINE", "ONLINE", or "DISABLED" COMET_PROJECT_NAME: (Optional): str - Comet.ml project name for experiments COMET_OFFLINE_DIRECTORY: (Optional): str - folder to use for saving offline experiments when `COMET_MODE` is "OFFLINE" For a number of configurable items in the environment, see `here `__ r ZONLINE project_nameZCOMET_PROJECT_NAMErgNz)Automatic Comet.ml online logging enabledZOFFLINEZCOMET_OFFLINE_DIRECTORYz./Zoffline_directoryzLAutomatic Comet.ml offline logging enabled; use `comet upload` when finished transformers) frameworkzargs/)prefixrpzconfig/) r1r2r3comet_mlZ Experimentr4r5ZOfflineExperimentZ_set_model_graphrZ_log_parametersrri)r:Z comet_moder experimentr;r;r<r6s    zTFTrainer.setup_comet)datasetrZrX descriptionprediction_loss_onlyr>cCsh|dk r |n|jj}td|dtd||dkrVtd|jj|td|jjd}d}|j|jjdkrd|_t |D]\}} | | } | \} } |st | t r| d} t | t r| d} |jj d krT| jD],} |dkr| }qtj|| dd }q| jD].} |dkr:| }ntj|| dd }q"nP|dkrh| }ntj|| dd }|dkr| }ntj|| dd }||d krqq|jdk r|dk r|dk r|t||d }ni}|j||d <t|D]&}|d s|||d |<q|jjrZt|drZt|dt|||dS)z Prediction/evaluation loop, shared by [`~TFTrainer.evaluate`] and [`~TFTrainer.predict`]. Works both with or without labels. Nz***** Running z *****z Num examples in dataset = Evaluationz' Num examples in used in evaluation = z Batch size = rrZaxis) predictions label_idsr*Zeval__past)ryrzr()rrvr4r5rWr* reset_states past_indexr{ enumeratedistributed_prediction_steps isinstancetuple n_replicasvaluesrFnpappendrrresultlistkeys startswithpophasattrdelattrr)r:rtrZrXrurvrzpredssteprJlogits_labelsvalr(keyr;r;r<prediction_loop1s^                   zTFTrainer.prediction_loop)logsr>c Cs|j|d<|jrV|j,|D]\}}tjj|||jdq$W5QRX|jt rlt j ||jdt rt j}|dk r|j||j|jdd|d|ji}t|dS)z Log `logs` on the various objects watching training. Subclass and override this method to inject custom behavior. Args: logs (`Dict[str, float]`): The values to log. epoch)rNro)rrrpr)r%r as_defaultitemsr&r.Zscalarr$flushr rllogrrrriZget_global_experimentZ _log_metricsr4r5)r:rkvrsoutputr;r;r<rs&     z TFTrainer.logcCsD||\}}}|j|||dd}|j}|j|d<|||jS)a Run evaluation and returns metrics. The calling script will be responsible for providing a method to compute metrics, as they are task-dependent (pass it to the init `compute_metrics` argument). Args: eval_dataset ([`~tf.data.Dataset`], *optional*): Pass a dataset if you wish to override `self.eval_dataset`. The dataset should yield tuples of `(features, labels)` where `features` is a dict of input features and `labels` is the labels. If `labels` is a tensor, the loss is calculated by the model by calling `model(features, labels=labels)`. If `labels` is a dict, such as when using a QuestionAnswering head model with multiple targets, the loss is instead calculated by calling `model(features, **labels)`. Returns: A dictionary containing the evaluation loss and the potential metrics computed from the predictions. rwrur)r[rr(r%r)r:rZeval_dsrZrXrrr;r;r<evaluates   zTFTrainer.evaluate)featuresrnb_instances_in_global_batchr>cCs6|||d\}}|tj||jd}|j||S)z Compute the prediction on features and update the loss with labels. Subclass and override to inject some custom behavior. Fdtype) run_modelr&castrr* update_state)r:rrrper_example_lossr scaled_lossr;r;r<prediction_steps zTFTrainer.prediction_stepcCs,||}|||}|jj|j|}|SN)_compute_nb_instances_get_step_inputsrrPrunr)r:rJnb_instances_in_batchinputsrr;r;r<rs  z&TFTrainer.distributed_prediction_stepsc CsL|}|jjr tjjddd|j|j|j }|jj rDt j nt j }||}t|d}||_|jjdkr|jj}|jj|jt|jj|jdk}n|j|jj}|jj}t|}|jjZ|j|dtj|jjt}tjj|j|jd}tjj |||jj!d|j_"|jj#}d} d} |jj"j$rt%&d|jj"j$d |'|jj"j$(|)|_*|j*|j} |j*|j} t%&d t%&d | t%&d |j*t%&d | dtjj+,|j*|j-.tj/d|j0W5QRX|j-1t%&dt%&d|jt%&d|t%&d|jj2t%&d|j t%&d|jj3t%&d|jt%&d|tj4j56|_7t8j89} t:| t|D]H} |jj;dkrd|_||)|_*| | d|j|_?|j7@| d}|jjrZi}|)|d<|j?|d<|A||j*dkr|jjr|j-.tjjBd|j*|jjCdW5QRX|jjDdkr|jjEtFjGkr|j*|jjDdkr|H|jjIdkr|j*|jjIdks|j*dkrH|jjJrHi}|)|d<|K|j*)|d<|j?|d<|A||jjLdkr|j*|jjLdkr|jj"M}t%&d|j*d||jjdkr|j*|krq|j*|jdkrqʐq|j7N|jjdkr|j*|jjkrqqt8j89}t%&d tO|| W5QRX|jj;rHtP|d!rHtQ|d!dS)"z2 Train method to train the model. T)graphZprofilerrrr^)r!r)Z max_to_keepzCheckpoint file z$ found and restoring from checkpointzE Continuing training from checkpoint, will skip to saved global_stepz! Continuing training from epoch z' Continuing training from global step z Will skip the first z steps in the first epochrz***** Running training *****z Num examples = z Num Epochs = z( Instantaneous batch size per device = zE Total train batch size (w. parallel, distributed & accumulation) = z Gradient Accumulation steps = z Steps per epoch = z Total optimization steps = Nlossrtraining)rjrZprofiler_outdirrezSaving checkpoint for step z at zTraining took: r{)RrSrdebugr&r.Ztrace_onr#resetrGrDrKrTrUrVmaxZsteps_per_epochZ max_stepsintZnum_train_epochsfloatrPscoperfr1pathjoin output_dirrtrainZ Checkpointr!rZCheckpointManagerZsave_total_limitZ ckpt_manager iterationsZlatest_checkpointr4r5restoreZexpect_partialrFr$rNZset_steprrtextZto_json_stringrZper_device_train_batch_sizerCr'r(r) train_lossdatetimenowranger}r{r~distributed_training_stepsr%rrZ trace_exportr/Z eval_stepsZevaluation_strategyrZSTEPSrZ logging_stepsZlogging_first_stepr"Z save_stepssaver|strrr)r:Ztrain_dsZnum_update_steps_per_epochrYZt_totalZepochsfolderZckptrZepochs_trainedZsteps_trained_in_current_epoch start_timeZ epoch_iterrrJZ training_lossrZckpt_save_pathend_timer;r;r<rs                         "   "     "zTFTrainer.traincCs|||d\}}|tj||jd}t||jj}ddt||jjD}|jj dkrd| ||j ||jj dkr|SdS)z Perform a training step on features and labels. Subclass and override to inject some custom behavior. TrcSs&g|]\}}|dk r|nt|qSr)r&Z zeros_like).0grr;r;r< sz+TFTrainer.training_step..rN) rr&rr gradientsrtrainable_variablesziprrCr#rr)r:rrrrrrrr;r;r< training_stepvs     zTFTrainer.training_stepcszjjdkr8|||}jtt|jjn>t jjD]}fdd| Dt |r|djj jjn*t|trfdd| Dntd|fdd| D}t |rt j|jj jjdgdd}qFt|tr.fd d| D}qFtdqFjj}fd d |D}jtt|jjjdS) Nrcs*i|]"\}}||djjjjqSrrrBrrrftr:r;r< sz-TFTrainer.apply_gradients..cs*i|]"\}}||djjjjqSrrrrZlblrr;r<rsz0The labels must be either a tf.Tensor or a dict.cs<i|]4\}}|tj|jjjjd|gddqSNrrxr&concatrrBrr)reduced_featuresr:r;r<rs rrxcs<i|]4\}}|tj|jjjjd|gddqSrrr)reduced_labelsr:r;r<rs cs$g|]}t|jj jjqSr;)r&Z clip_by_valuerZ max_grad_norm)rZgradrr;r<rsz-TFTrainer.apply_gradients..)rrCrr!apply_gradientsrrrrr&rrZ is_tensorrBrrdictrArr#rr)r:rrrrrr;)rrr:r<rsD           zTFTrainer.apply_gradientsc CsD|jj.||}|||}|jj|j|W5QRXdSr)rrPrrrrr)r:rJrrr;r;r<rs  z$TFTrainer.distributed_training_stepscCs@|d}t|tr"tj|jdd}ttj|dktjd}|S)Nrrxir)rr r&rrZ reduce_sumrZint32)rJr nb_instancesr;r;r<rs  zTFTrainer._compute_nb_instancescCs4|\}}t|tr&t|gt|j}|||f}|Sr)rr lenr)rJrrrZ step_inputsr;r;r<rs   zTFTrainer._get_step_inputscCs|jjdkr&t|dddk r&|j|d<t|trP|j|fd|i|dd}n|j|||ddd}|dd\}}|jjdkr||jj|_||fS)a Computes the loss of the given features and labels pair. Subclass and override this method if you want to inject some custom behavior. Args: features (`tf.Tensor`): A batch of input features. labels (`tf.Tensor`): A batch of labels. training (`bool`): Whether or not to run the model in training mode. Returns: A tuple of two `tf.Tensor`: The loss and logits. rr{NZmemsr)rr)rr}getattrr{rrr)r:rrroutputsrrr;r;r<rs    zTFTrainer.run_modelcCs"||\}}}|j|||ddS)a Run prediction and returns predictions and potential metrics. Depending on the dataset and your use case, your test dataset may contain labels. In that case, this method will also return metrics, like in `evaluate()`. Args: test_dataset ([`~tf.data.Dataset`]): Dataset to run the predictions on. The dataset should yield tuples of `(features, labels)` where `features` is a dict of input features and `labels` is the labels. If `labels` is a tensor, the loss is calculated by the model by calling `model(features, labels=labels)`. If `labels` is a dict, such as when using a QuestionAnswering head model with multiple targets, the loss is instead calculated by calling `model(features, **labels)` Returns: *NamedTuple* A namedtuple with the following keys: - predictions (`np.ndarray`): The predictions on `test_dataset`. - label_ids (`np.ndarray`, *optional*): The labels (if the dataset contained some). - metrics (`Dict[str, float]`, *optional*): The potential dictionary of metrics (if the dataset contained labels). Z Predictionr)r]r)r:r\Ztest_dsrZrXr;r;r<predictszTFTrainer.predict)rcCsH|dk r |n|jj}td|t|jts8td|j|dS)zV Will save the model, so you can reload it using `from_pretrained()`. NzSaving model in z1Trainer.model appears to not be a PreTrainedModel) rrr4r5rrr rAZsave_pretrained)r:rr;r;r< save_models  zTFTrainer.save_model)NNNNr)N)N)N)N)2__name__ __module__ __qualname____doc__r rrr&rMZDatasetrrrr.Z SummaryWriterrr'rZ OptimizerZ schedulesZLearningRateScheduler=rSr[r]rrfr0r6rboolrrrrrZTensorrfunctionrrrrr staticmethodrrrrrr;r;r;r<r=sh$    7!$ V "  5   r)*rrrTr1r+typingrrrrutilsrZ integrationsrr rFrZ tensorflowr&Z#tensorflow.python.distribute.valuesr Zmodeling_tf_utilsr Zoptimization_tfr r Z trainer_utilsrrrrrrZtraining_args_tfrrrlrrZ get_loggerrr4rr;r;r;r<s*