U ,-e@sbdZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlZddlZddlmZddlmZddlmZmZmZmZmZmZmZmZddlmZmZm Z ddl!m"Z#ddl$Z%ddl&Z&ddl'm(Z)ddl*m+Z+m,Z,m-Z-dd l.m/Z/dd l&m0Z0dd l1m2Z2m3Z3m4Z4m5Z5dd l6m7Z7dd l8m9Z9ddl:m;Z;mm?Z?m@Z@ddlAmBZBddlCmDZDmEZEddlFmGZGmHZHmIZIddlJmKZKddlLmMZMmNZNmOZOddlPmQZQmRZRddlSmTZTmUZUddlVmWZWmXZXddlYmZZZddl[m\Z\m]Z]m^Z^m_Z_m`Z`maZambZbddlcmdZdmeZemfZfmgZgmhZhmiZimjZjmkZkmlZlmmZmmnZnmoZompZpmqZqmrZrmsZsmtZtmuZuddlvmwZwmxZxmyZymzZzm{Z{m|Z|m}Z}m~Z~mZmZmZmZmZmZmZmZmZmZmZmZmZmZmZddlmZmZmZddl"mZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZddlmZe]gZe_ZerddlmZeZerdd lmZerddlZed!d"rddlmmZddlmmZe r.eBd#ddlZdd$lmZdd%lmZdd&lmZdd'lmZdd(lmZerddlmm&Zdd lm7Ze/e͡e/d)kZdd*lcmZmZmZmZnd!ZerddlZerdd+lmZer dd,lmZmZdd lm7Zdd-lmZmZe/eۡe/d.kr dd/lmZmZmZmZeIr dd0lmZer.ddlZeeZd1Zd2Zd3Zd4Zd5Zd6ZGd7d8d8ZdS)9uc The Trainer class, to easily train a 🤗 Transformers from scratch or finetune it on a new task. N)Mapping)Path) TYPE_CHECKINGAnyCallableDictListOptionalTupleUnion)#get_reporting_integration_callbacks hp_paramsis_fairscale_available) Repository create_repo upload_folder)version)nn) DataLoaderDataset RandomSamplerSequentialSampler) __version__)PretrainedConfig) DataCollatorDataCollatorWithPaddingdefault_data_collator) DebugOptionDebugUnderflowOverflow)dep_version_check)"ALL_HYPERPARAMETER_SEARCH_BACKENDSdefault_hp_search_backend)deepspeed_initdeepspeed_load_checkpointis_deepspeed_available)TrainingSummary)PreTrainedModelload_sharded_checkpoint unwrap_model)!MODEL_FOR_CAUSAL_LM_MAPPING_NAMESMODEL_MAPPING_NAMES) Adafactor get_scheduler)ALL_LAYERNORM_LAYERSis_torch_less_than_1_11)PreTrainedTokenizerBase)CallbackHandlerDefaultFlowCallbackPrinterCallbackProgressCallbackTrainerCallbackTrainerControl TrainerState)DistributedTensorGathererIterableDatasetShard LabelSmootherLengthGroupedSamplerSequentialDistributedSamplerdistributed_broadcast_scalarsdistributed_concatfind_batch_sizeget_dataloader_samplerget_model_param_countget_module_class_from_nameget_parameter_names nested_concat nested_detachnested_numpifynested_xla_mesh_reducereissue_pt_warningsremove_dummy_checkpoint)PREFIX_CHECKPOINT_DIRBestRunEvalLoopOutputEvalPrediction FSDPOptionHPSearchBackend HubStrategyIntervalStrategyPredictionOutputRemoveColumnsCollatorShardedDDPOptionTrainerMemoryTracker TrainOutputdefault_compute_objectivedenumpify_detensorizeenable_full_determinismfind_executable_batch_sizeget_last_checkpoint has_lengthnumber_of_arguments seed_workerset_seed speed_metrics)OptimizerNames ParallelModeTrainingArguments)ADAPTER_CONFIG_NAMEADAPTER_SAFE_WEIGHTS_NAMEADAPTER_WEIGHTS_NAME CONFIG_NAMESAFE_WEIGHTS_INDEX_NAMESAFE_WEIGHTS_NAMEWEIGHTS_INDEX_NAME WEIGHTS_NAMEPushInProgresscan_return_loss find_labelsis_accelerate_availableis_apex_availableis_bitsandbytes_availableis_datasets_availableis_in_notebookis_ipex_availableis_peft_availableis_safetensors_availableis_sagemaker_dp_enabledis_sagemaker_mp_enabledis_torch_compile_availableis_torch_neuroncore_availableis_torch_tpu_availablelogging strtobool)QuantizationMethod)NotebookProgressCallback)ampF)Z check_device fairscale)FullyShardedDataParallel)ShardedDataParallel) auto_wrap)OSS)ShardedGradScalerz1.10)smp_forward_backwardsmp_forward_only smp_gathersmp_nested_concat) PeftModel) Acceleratorskip_first_batches)DistributedDataParallelKwargsGradientAccumulationPlugin0.20.3)load_fsdp_modelload_fsdp_optimizersave_fsdp_modelsave_fsdp_optimizer)DeepSpeedSchedulerWrapperztraining_args.binztrainer_state.jsonz optimizer.ptz optimizer.binz scheduler.ptz scaler.ptc@seZdZdZddlmZmZmZmZm Z de e e j feeeeeee eeeeffeeeege feeegefeeeeejjejjjfeeejejgejfd ddZ d d Z!d d Z"d dZ#ddZ$ddZ%ddeedddZ&deeeedddZ'eej(j)j*dddZ+e,dddZ-eeej(j)j*dd d!Z.deee,dd"d#Z/ee,d$d%d&Z0e1d'd(d)Z2eedd*d+Z3d,d-Z4e5eee6e6fd.d/d0Z7de1ejjd1d2d3Z8e,e1d4d5d6Z9de,ee1e1d7d8d9Z:e d:eee6ffd;dd?d@Z=dAdBZ>ddCdDZ?ddFdGZ@dEejAfdHdIZBddKdLZCdee eeDfe d:eee6ffeeedMdNdOZEddPdQZFdRdSZGddTdUZHdVdWZIdXdYZJdZd[ZKd\d]ZLdd^d_ZMd`daZNdeed:geee [`Trainer`] is optimized to work with the [`PreTrainedModel`] provided by the library. You can still use your own models defined as `torch.nn.Module` as long as they work the same way as the 🤗 Transformers models. args ([`TrainingArguments`], *optional*): The arguments to tweak for training. Will default to a basic instance of [`TrainingArguments`] with the `output_dir` set to a directory named *tmp_trainer* in the current directory if not provided. data_collator (`DataCollator`, *optional*): The function to use to form a batch from a list of elements of `train_dataset` or `eval_dataset`. Will default to [`default_data_collator`] if no `tokenizer` is provided, an instance of [`DataCollatorWithPadding`] otherwise. train_dataset (`torch.utils.data.Dataset` or `torch.utils.data.IterableDataset`, *optional*): The dataset to use for training. If it is a [`~datasets.Dataset`], columns not accepted by the `model.forward()` method are automatically removed. Note that if it's a `torch.utils.data.IterableDataset` with some randomization and you are training in a distributed fashion, your iterable dataset should either use a internal attribute `generator` that is a `torch.Generator` for the randomization that must be identical on all processes (and the Trainer will manually set the seed of this `generator` at each epoch) or have a `set_epoch()` method that internally sets the seed of the RNGs used. eval_dataset (Union[`torch.utils.data.Dataset`, Dict[str, `torch.utils.data.Dataset`]), *optional*): The dataset to use for evaluation. If it is a [`~datasets.Dataset`], columns not accepted by the `model.forward()` method are automatically removed. If it is a dictionary, it will evaluate on each dataset prepending the dictionary key to the metric name. tokenizer ([`PreTrainedTokenizerBase`], *optional*): The tokenizer used to preprocess the data. If provided, will be used to automatically pad the inputs to the maximum length when batching inputs, and it will be saved along the model to make it easier to rerun an interrupted training or reuse the fine-tuned model. model_init (`Callable[[], PreTrainedModel]`, *optional*): A function that instantiates the model to be used. If provided, each call to [`~Trainer.train`] will start from a new instance of the model as given by this function. The function may have zero argument, or a single one containing the optuna/Ray Tune/SigOpt trial object, to be able to choose different architectures according to hyper parameters (such as layer count, sizes of inner layers, dropout probabilities etc). compute_metrics (`Callable[[EvalPrediction], Dict]`, *optional*): The function that will be used to compute metrics at evaluation. Must take a [`EvalPrediction`] and return a dictionary string to metric values. callbacks (List of [`TrainerCallback`], *optional*): A list of callbacks to customize the training loop. Will add those to the list of default callbacks detailed in [here](callback). If you want to remove one of the default callbacks used, use the [`Trainer.remove_callback`] method. optimizers (`Tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR]`, *optional*): A tuple containing the optimizer and the scheduler to use. Will default to an instance of [`AdamW`] on your model and a scheduler given by [`get_linear_schedule_with_warmup`] controlled by `args`. preprocess_logits_for_metrics (`Callable[[torch.Tensor, torch.Tensor], torch.Tensor]`, *optional*): A function that preprocess the logits right before caching them at each evaluation step. Must take two tensors, the logits and the labels, and return the logits once processed as desired. The modifications made by this function will be reflected in the predictions received by `compute_metrics`. Note that the labels (second parameter) will be `None` if the dataset does not have them. Important attributes: - **model** -- Always points to the core model. If using a transformers model, it will be a [`PreTrainedModel`] subclass. - **model_wrapped** -- Always points to the most external model in case one or more other modules wrap the original model. This is the model that should be used for the forward pass. For example, under `DeepSpeed`, the inner model is wrapped in `DeepSpeed` and then again in `torch.nn.DistributedDataParallel`. If the inner model hasn't been wrapped, then `self.model_wrapped` is the same as `self.model`. - **is_model_parallel** -- Whether or not a model has been switched to a model parallel mode (different from data parallelism, this means some of the model layers are split on different GPUs). - **place_model_on_device** -- Whether or not to automatically place the model on the device - it will be set to `False` if model parallel or deepspeed is used, or if the default `TrainingArguments.place_model_on_device` is overridden to return `False` . - **is_in_train** -- Whether or not a model is currently running `train` (e.g. when `evaluate` is called while in `train`) r )_get_learning_rate log_metricsmetrics_format save_metrics save_stateNNN) modelargs data_collator train_dataset eval_dataset tokenizer model_initcompute_metrics callbacks optimizerspreprocess_logits_for_metricsc Cs |dkr(d} td| dt| d}||_|jjrBt|jjn t|jjd|_d|_ d|_ | t |jj |_|j|} t| |j|dkr|dk r||_|}qtdn|dk rtdt||_|jjtkrtd|jjd t|d r |jr |j r d |_!nd|_!t"|d ddk rd dt#|j$%D}t&|dkrfd |_!n.t&|dkr|jj't('|dk|_!nd|_!|j!rtdt)ot*|t+}t"|ddot"|dd }|r|stdn|rt"|ddstdd|_,t&|j,dkr|j-r,tdt&|j.dkrDtd|j/t0j1kr\tdnt2snt3dnrt4j5|j,krt6dkrt3dt7j8dnFt4j5|j,krt4j5|_,n.t4j9|j,krt4j9|_,nt4j:|j,krt4j:|_,d|_.t&|j.dkr|j-rtd|j;ds(|j/t0j1kr(tdt<=t<=t(j8j>t<=d krPtd!dd"l?m@}mA}tBjC|j.krx|jC|_.n.tBjD|j.kr|jD|_.ntBjE|j.kr|jE|_.|jF|_Gd#|jj;krd$|jj;Hd#gkr|jI|_Gd|_J|jj;Hd%drd |_J|jK|_K|j!sP|j-sP|jLs |jMr(|jNrP|j,t4j9t4j:fksP|j.dk sP|jOrVd|_K|dkrdtPntQ|}|dk rz|n||_R||_S||_T||_U|jKrt"|d&dtVjWks|X||j'|j!rd|j_Y||_Z||_[||_\| |_]| \|_^|__|dk r|j^dk s|j_dk rtd't`r|j^dk r|j[aD]}|j'}qLq8|j^jbD],}t&|d(dkrT|d(dj'}qqT||krtd)|j,dk s|j-s|j.dk r|j^dk s|j_dk rtd*tctd|jje}| dkr|n|| } tf| |j[|jU|j^|j_|_g|h|jjir&tjntkd|_ld|_m|jjnrJ|o|jjprftqjr|jjsd d+tt|jRsttt"|jRd,drtd-|judkrtd.|dk rtv|s|judkrtd/|dk rt*|t(jwjxjyr|jzrtd0d|_{d|_|d|_}d|_~tr|jr(td1trx|jtjjjkrtd2tjjjd3|jd4tjjjtjjj|_n(ttjjd5rtd2tjjjd6|js|jr|j,dk r|jd7kr|j't('d8kr|jrtd9nd:|_nd;|_td<|jd=d|_|js$|jr|j-sts|j,dk r|jd;krd |_}|jr`t(jnt(j|_|jt(jk|_|jrt|_n|jd:krd |_~t(j|_n"|jd>krtst3d?d |_|tr|j}r|jdk r|jdkrtd@|jjdk rt|jjdA|_nd|_t||dB|_t|_d|_d|_d|_t|j[j}|jjdk rj|n|jj|_t|j[j|_|jg|j|j|j|_|j|_d|_|j|j rt stdCdS)DNZ tmp_trainerz1No `TrainingArguments` passed, using `output_dir=z`. output_dirFz<`Trainer` requires either a `model` or `model_init` argumentz`Trainer` requires either a `model` or `model_init` argument, but not both. `model_init` will overwrite your model when calling the `train` method. This will become a fatal error in the next release.zThe model you have picked (a) cannot be used as is for training: it only computes hidden states and does not accept any labels. You should choose a model with a head suitable for your task like any of the `AutoModelForXxx` listed at https://huggingface.co/docs/transformers/model_doc/autois_parallelizableT hf_device_mapcSsg|]}|dkr|qS))cpuZdisk).0devicerrU/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/trainer.py sz$Trainer.__init__..r rzYou have loaded a model on multiple GPUs. `is_model_parallel` attribute will be force-set to `True` to avoid any unexpected behavior such as device placement mismatching.Z is_quantizedZ_hf_peft_config_loadedzYou cannot perform fine-tuning on purely quantized models. Please attach trainable adapters on top of the quantized model to correctly perform fine-tuning. Please see: https://huggingface.co/docs/transformers/peft for more detailsZ_is_quantized_training_enabledzThe model you want to train is loaded in 8-bit precision. if you want to fine-tune an 8-bit model, please make sure that you have installed `bitsandbytes>=0.37.0`. zaUsing --sharded_ddp xxx together with --deepspeed is not possible, deactivate one of those flags.z\Using --sharded_ddp xxx together with --fsdp is not possible, deactivate one of those flags.z5Using sharded DDP only works in distributed training.zASharded DDP training requires fairscale: `pip install fairscale`.zZSharded DDP in a mode other than simple training requires fairscale version >= 0.3, found zD. Upgrade your fairscale library: `pip install --upgrade fairscale`.zZUsing --fsdp xxx together with --deepspeed is not possible, deactivate one of those flags.xlaz.Using fsdp only works in distributed training.z1.12.0zFSDP requires PyTorch >= 1.12.0)BackwardPrefetchShardingStrategybackward_prefetchZ backward_postlimit_all_gathersZquantization_methodzPassing a `model_init` is incompatible with providing the `optimizers` argument. You should subclass `Trainer` and override the `create_optimizer_and_scheduler` method.paramsa[The model and the optimizer parameters are not on the same device, which probably means you created an optimizer around your model **before** putting on the device and passing it to the `Trainer`. Make sure the lines `import torch_xla.core.xla_model as xm` and `model.to(xm.xla_device())` is performed before the optimizer creation in your script.zPassing `optimizers` is not allowed if Fairscale, Deepspeed or PyTorch FSDP is enabled.You should subclass `Trainer` and override the `create_optimizer_and_scheduler` method.exist_okZ collate_batchzRThe `data_collator` should be a simple callable (function, class with `__call__`).zHmax_steps is given, it will override any value given in num_train_epochszThe train_dataset does not implement __len__, max_steps has to be specified. The number of steps needs to be known in advance for the learning rate scheduler.zTthe `--group_by_length` option is only available for `Dataset`, not `IterableDatasetzOSageMaker Model Parallelism does not support BF16 yet. Please use FP16 instead z(FP16 provided in SM_HP_MP_PARAMETERS is z*,but FP16 provided in trainer argument is z ,setting to fp16zJ, but SageMaker Model Parallelism < 1.10 does not support FP16 in trainer.autorz2Tried to use `fp16` but it is not supported on cpuZcpu_ampZcuda_ampzUsing z half precision backendapexzcUsing FP16 with APEX but APEX is not installed, please refer to https://www.github.com/nvidia/apex.zSageMaker Model Parallelism in mixed precision mode does not support gradient clipping yet. Pass along 'max_grad_norm': 0 in your hyperparameters.)epsilon)is_local_process_zerois_world_process_zeroz3Using torch.compile requires PyTorch 2.0 or higher.)loggerinforcrfull_determinismrYseedr_hp_name deepspeed is_in_train"create_accelerator_and_postprocessrUZskip_memory_metrics_memory_trackerstartZget_process_log_levelr| set_verbosityZ_setup_devicesrcall_model_init RuntimeErrorwarningswarn FutureWarning __class____name__r+ ValueErrorhasattrrZmodel_parallelZis_model_parallelgetattrsetrvalueslenrtorchru isinstancer sharded_ddpis_deepspeed_enabledfsdp parallel_moderb DISTRIBUTEDr ImportErrorrTSIMPLEFullyShardedDDPrr ZERO_DP_2 ZERO_DP_3 fsdp_configrparse base_versionZ2torch.distributed.fsdp.fully_sharded_data_parallelrrrNZ FULL_SHARDZ SHARD_GRAD_OPZNO_SHARDZ BACKWARD_PRErgetZ BACKWARD_POSTrplace_model_on_devicefp16_full_evalbf16_full_evaldo_trainis_fsdp_enabledrrrrrrr~ZBITS_AND_BYTES_move_model_to_deviceZ_n_gpu model_wrappedrrr optimizer lr_schedulerr{ parametersZ param_groupsDEFAULT_CALLBACKSr Z report_tor1callback_handler add_callbackZ disable_tqdmr3DEFAULT_PROGRESS_CALLBACKZ_loggers_initialized hub_model_id push_to_hub init_hf_repo should_saveosmakedirsrcallable max_stepsr\utilsdataIterableDatasetgroup_by_length_signature_columnsuse_apex use_cuda_amp use_cpu_amprxbf16IS_SAGEMAKER_MP_POST_1_10rsmpstatecfgwarningZhalf_precision_backenddo_grad_scalingfloat16bfloat16 amp_dtyperscalerrp max_grad_normZlabel_smoothing_factorr:label_smootherr7rrr6control current_floshp_search_backenduse_tune_checkpointsrn label_namesrmZ on_init_endtrain_batch_size_train_batch_size_created_lr_schedulerstop_and_update_metricsZ torch_compilery)selfrrrrrrrrrrrr log_levelZdevicesZ_is_peft_modelZ_is_quantized_and_base_modelrrZdefault_collatorparamZ model_deviceZ param_groupZoptimizer_deviceZdefault_callbacksZdefault_label_namesrrr__init__Bs                   "           "           zTrainer.__init__cCs|j|dS)ac Add a callback to the current list of [`~transformer.TrainerCallback`]. Args: callback (`type` or [`~transformer.TrainerCallback`]): A [`~transformer.TrainerCallback`] class or an instance of a [`~transformer.TrainerCallback`]. In the first case, will instantiate a member of that class. N)rrrcallbackrrrrs zTrainer.add_callbackcCs |j|S)aF Remove a callback from the current list of [`~transformer.TrainerCallback`] and returns it. If the callback is not found, returns `None` (and no error is raised). Args: callback (`type` or [`~transformer.TrainerCallback`]): A [`~transformer.TrainerCallback`] class or an instance of a [`~transformer.TrainerCallback`]. In the first case, will pop the first member of that class found in the list of callbacks. Returns: [`~transformer.TrainerCallback`]: The callback removed, if found. )r pop_callbackr rrrr"szTrainer.pop_callbackcCs|j|dS)a Remove a callback from the current list of [`~transformer.TrainerCallback`]. Args: callback (`type` or [`~transformer.TrainerCallback`]): A [`~transformer.TrainerCallback`] class or an instance of a [`~transformer.TrainerCallback`]. In the first case, will remove the first member of that class found in the list of callbacks. N)rremove_callbackr rrrr#s zTrainer.remove_callbackcCs.||}|jjtjkr*t|dr*|dS)N tie_weights)torrrbZTPUrr$)rrrrrrrs zTrainer._move_model_to_devicecCsL|jdkrHt|jj}t|j|_|jttddg|j 7_dS)Nlabel label_ids) rinspect signaturerforwardlistrkeysrr)rr)rrr _set_signature_columns_if_neededs z(Trainer._set_signature_columns_if_neededzdatasets.Dataset)dataset descriptionc s|jjs S||j}ttjt|}t|dkr|dkrHdn d|d}t d|d|j j j dd |d d |d |j j j d fd d |D}ttjtdkrވjjd|jddS|SdS)Nrzin the z setzThe following columns z) don't have a corresponding argument in `z!.forward` and have been ignored: , z. If z are not expected by `z/.forward`, you can safely ignore this message.csg|]}|jkr|qSr) column_namesrkr.rrrs z2Trainer._remove_unused_columns..z1.4.0type format_kwargs)r6columnsr7)rremove_unused_columnsr-rr+rr2rrrrrrjoinrrdatasetsrZ set_formatformatZremove_columns)rr.r/signature_columnsZignored_columnsZdset_descriptionr8rr5r_remove_unused_columnss& :zTrainer._remove_unused_columns)rr/returncCs6|jjs |S||j}t||t||jjjd}|S)z=Wrap the data collator in a callable removing unused columns.)rr=rr/ model_name) rr9r-rrSrrrr)rrr/r=Zremove_columns_collatorrrr"_get_collator_with_removed_columns sz*Trainer._get_collator_with_removed_columns)r?cCs|jdkst|jsdS|jjrtrXt|jtjrX|jj|jj krR|j|jjnd}nd}|j dk rr|j j dnd}t |jj |jj|j||dSt|jSdS)Nr)r.lengthsmodel_input_name)rr\rrrrrr;rZlength_column_namer2rZmodel_input_namesr;rgradient_accumulation_stepsr)rrBrCrrr_get_train_samplers"zTrainer._get_train_samplercCs|jdkrtd|j}|j}tr@t|tjr@|j|dd}n|j|dd}|j ||j j |j j d}t|t jjjs||d<|j j|d<t|d<|jt|f|S) a@ Returns the training [`~torch.utils.data.DataLoader`]. Will use no sampler if `train_dataset` does not implement `__len__`, a random sampler (adapted to distributed training if necessary) otherwise. Subclass and override this method if you want to inject some custom behavior. Nz+Trainer: training requires a train_dataset.trainingr/ batch_sizeZ collate_fnZ num_workersZ pin_memorysampler drop_lastZworker_init_fn)rrrrrrr;rr>rArrdataloader_num_workersdataloader_pin_memoryrrrrrEdataloader_drop_lastr^ acceleratorpreparer)rrrdataloader_paramsrrrget_train_dataloader5s"   zTrainer.get_train_dataloader)rr?cCsj|jjrNtr$t|ttdStrFt|t t |jj dSt |S|jj dkrbt |SdSdS)N) num_replicasrank)rSrTrIr )ruse_legacy_prediction_loopr{r<xmxrt_world_sizeZ get_ordinalrxrZdp_sizeZdp_rankZper_device_eval_batch_sizer world_size)rrrrr_get_eval_samplerVs$ zTrainer._get_eval_samplercCs|dkr|jdkrtd|dk r&|n|j}|j}trTt|tjrT|j|dd}n|j|dd}|j j ||j j |j j d}t|t jjjs|||d<|j j|d<|jt|f|S)a Returns the evaluation [`~torch.utils.data.DataLoader`]. Subclass and override this method if you want to inject some custom behavior. Args: eval_dataset (`torch.utils.data.Dataset`, *optional*): If provided, will override `self.eval_dataset`. If it is a [`~datasets.Dataset`], columns not accepted by the `model.forward()` method are automatically removed. It must implement `__len__`. Nz-Trainer: evaluation requires an eval_dataset.Z evaluationrGrHrJrK)rrrrrrr;rr>rAreval_batch_sizerLrMrrrrrYrNrOrPr)rrrrQrrrget_eval_dataloaderls  zTrainer.get_eval_dataloader) test_datasetr?cCs|j}tr(t|tjr(|j|dd}n|j|dd}|jj||jj |jj d}t|t j j jsz|||d<|jj|d<|jt|f|S)a Returns the test [`~torch.utils.data.DataLoader`]. Subclass and override this method if you want to inject some custom behavior. Args: test_dataset (`torch.utils.data.Dataset`, *optional*): The test dataset to use. If it is a [`~datasets.Dataset`], columns not accepted by the `model.forward()` method are automatically removed. It must implement `__len__`. testrGrHrJrK)rrrrr;rr>rArrZrLrMrrrrrYrNrOrPr)rr\rrQrrrget_test_dataloaders  zTrainer.get_test_dataloadernum_training_stepscCs8|tr tjjjr |jj}n|j}|j||ddS)aZ Setup the optimizer and the learning rate scheduler. We provide a reasonable default that works well. If you want to use something else, you can pass a tuple in the Trainer's init through `optimizers`, or subclass and override this method (or `create_optimizer` and/or `create_scheduler`) in a subclass. r`rN)create_optimizerrrr r rrcreate_schedulerrr`rrrrcreate_optimizer_and_schedulers  z&Trainer.create_optimizer_and_schedulercCst|t}dd|D}|S)a! Get all parameter names that weight decay will be applied to Note that some models implement their own layernorm instead of calling nn.LayerNorm, weight decay could still apply to those modules since this function only filter out instance of nn.LayerNorm cSsg|]}d|kr|qS)Zbiasrrnamerrrrsz5Trainer.get_decay_parameter_names..)rCr.)rrdecay_parametersrrrget_decay_parameter_namess z!Trainer.get_decay_parameter_namesc sttr |jn|j}|jdkrX||fdd|D|jjdfdd|Dddg}t |j\}}|j t j krt f||d||_n||f||_|jdkrXd dl}|jj}d }|D]n}t|tjr|td d |D7}td |d |dd||ddditd|dqtd|ddtrnt|j|_|jS)a Setup the optimizer. We provide a reasonable default that works well. If you want to use something else, you can pass a tuple in the Trainer's init through `optimizers`, or subclass and override this method in a subclass. Ncs"g|]\}}|kr|jr|qSrZ requires_gradrnprhrrrsz,Trainer.create_optimizer..)r weight_decaycs"g|]\}}|kr|jr|qSrrjrkrnrrrs)roptimZAdam8bitrcSsi|]}||qSr)Zdata_ptrnumel)rrmrrr sz,Trainer.create_optimizer..zskipped z: izM paramsweight optim_bits zbitsandbytes: will optimize z in fp32z skipped: ) rxrrrriZnamed_parametersrrorget_optimizer_cls_and_kwargsrrTrrr bitsandbytesrqZGlobalOptimManagerZ get_instancemodulesrrZ EmbeddingsumrrrrZregister_module_overridedebugrZDistributedOptimizer) rZ opt_modelZoptimizer_grouped_parameters optimizer_clsoptimizer_kwargsrxmanagerZskippedmodulerrnrrbsJ          zTrainer.create_optimizer)rr?c Csi}|jr:|jdddD]}|d\}}|||<qd|ji}|j|jf|jd}|jtj kr|t }| dddn|jtj krd d l m}|}| |n|jtjtjfkrd d lm}|}| ||jtjkr| d d in|jtjkr@zd d lm}|}| |Wntk r:tdYnXnR|jtjkrzd dlm} | }| |Wntk rtdYnXn|jtjtjtjtjtjtjtjtj fkrzd dl!m}m"} d} d} d}|} d|jkrd } d|jkrd} d|jkr|}n d|jkr0| }d|j|jfi} | | d}| | | |Wntk rntdYnXt#rt$%t&j'$dt$%dkrt()dn|jtj*kr:zhd dl+m,}|}| || t-|.d d!t/t0|.d"d#t/t0|.d$d#t/t0|.d%d&d'Wntk r6td(YnXnX|jtj1krRt0jj1}n@|jtj2krjt0jj3}n(|jtj4krt0jj5}ntd)|j||fS)*z Returns the optimizer class and optimizer parameters based on the training arguments. Args: args (`transformers.training_args.TrainingArguments`): The training arguments for the training session.  r0,=lr)betasZepsF)Zscale_parameterZ relative_stepr )AdamWrZfusedTz7Trainer failed to import syncfree AdamW from torch_xla.) FusedAdamzFTrainer tried to instantiate apex FusedAdam but apex is not installed!)rLionrvNZpaged8bitZadamZlionr)is_pagedruzDTrainer tried to instantiate bnb optimizer but bnb is not installed!rxz0.41.1zYou are using 8-bit optimizers with a version of `bitsandbytes` < 0.41.1. It is recommended to update your version as a major bug has been fixed in 8-bit optimizers.)AnyPrecisionAdamWuse_kahan_summationFalsemomentum_dtypefloat32variance_dtypecompensation_buffer_dtyper)rrrrz4Please install https://github.com/pytorch/torchdistxz2Trainer cannot instantiate unsupported optimizer: )6 optim_argsreplacesplit learning_rateZ adam_beta1Z adam_beta2Z adam_epsilonrqraZ ADAFACTORr,updateZADAMW_HF optimizationrZ ADAMW_TORCHZADAMW_TORCH_FUSEDZ torch.optimZADAMW_TORCH_XLAZtorch_xla.amp.syncfreerrZADAMW_APEX_FUSEDZapex.optimizersrZ ADAMW_BNBZ ADAMW_8BITZ PAGED_ADAMWZPAGED_ADAMW_8BITZLIONZ LION_8BITZ PAGED_LIONZPAGED_LION_8BITZbitsandbytes.optimrrqrr importlibmetadatarr ZADAMW_ANYPRECISIONZtorchdistx.optimizersrr}rrrZSGDZADAGRADZAdagradZRMSPROPZRMSprop)rrmappingkeyvaluer}Z adam_kwargsr|rrrrruZadditional_optim_kwargsZ bnb_kwargsrrrrrws                            z$Trainer.get_optimizer_cls_and_kwargsracCsB|jdkr.zTrying to set zY in the hyperparameter search but there is no corresponding field in `TrainingArguments`.zTrial: zSigOpt Assignments: zW&B Sweep parameters: z7For sweeps with deepspeed, `args.deepspeed` must be setr)DeepSpeedPluginHfTrainerDeepSpeedConfig) hf_ds_config)_trialrrOOPTUNAhp_spaceRAYpopSIGOPT assignmentsitemsWANDBrrrr rr6setattrrrrrraccelerate.utilsr#transformers.integrations.deepspeedrhf_deepspeed_configtrainer_config_processdeepspeed_pluginr)rrrrrold_attrrrrrr_hp_search_setupsJ          zTrainer._hp_search_setup)rrmetricscCs|jdks|dkrdS|||_|jtjkrzddl}|js| |j|| r|j |j |j|j|n>|jtjkrddlm}|jjr||j fd|ji|dS)Nrtune objective)rcompute_objectivecopyrrOroptunaZstudyZ_is_multi_objectivereportZ should_pruner on_train_endrr rZ TrialPrunedrrayrr_tune_save_checkpoint)rrrrrrrrr_report_to_hp_searchs     zTrainer._report_to_hp_searchc Csddlm}|jsdS|j|jjd}tj|t d|jj}|j |dd|j j r|j tj|tt|jtj|tt|jtj|tW5QRXdS)Nrr)r-T_internal_call)rrrcheckpoint_dirr  global_steprpathr:rJ save_modelrr save_to_jsonTRAINER_STATE_NAMErsaver state_dictOPTIMIZER_NAMErSCHEDULER_NAME)rrrrrrrrs zTrainer._tune_save_checkpointcCsLt|j}|dkr|}n|dkr0||}ntd|dkrHtd|S)Nrr z'model_init should have 0 or 1 argument.z"model_init should not return None.)r]rr)rrZmodel_init_argcountrrrrrs   zTrainer.call_model_initFc s|s|dkrtd|Stt||z>t|}||jdd}|r`||_ |j j ddt ttt jjtdkrttrt jj|dd}n t jj|fddDdd}n@g}D]}t |}||qt|}t jj||dd }W5QRXW5QRXt j|}t |f|fW5QRX|}d|_d|_Wn@ttttt fk r} ztd | d W5d} ~ XYnX|S) NzAfailed to use PyTorch jit mode due to current dataloader is none.Z_original_forwardF cache_enabledz2.0.0)Zexample_kwarg_inputsstrictcsi|]}||qSrr)rrZ example_batchrrrs sz0Trainer.torch_jit_model_eval..rz'failed to use PyTorch jit mode due to: .)!rr nextiter_prepare_inputsreval__dict__rr*rOautocastrno_gradrrrrrdictZjittraceZ ones_likeappendtuplefreezerrrrrr IndexError) rrrrFZ jit_modelZoriginal_forwardZ jit_inputsrZexample_tensorerrrtorch_jit_model_eval sJ      &    $zTrainer.torch_jit_model_evalcCststdddl}|sT||js6|jjr6tjn|}|j ||dd|j d}n*|j sb| |j |||j ddd\}|_ |S)NzUsing IPEX but IPEX is not installed or IPEX's version does not match current PyTorch, please refer to https://github.com/intel/intel-extension-for-pytorch.rZO1F)dtypelevelZconv_bn_foldinginplaceT)rrrr) rtrZintel_extension_for_pytorchrrrrrroptimizerFtrainr)rrrFrZipexrrripex_optimize_model6s& zTrainer.ipex_optimize_modelTcs|jjr*|jrtjntj}|j|||d}trXt|j t j j rF|j St j ||jj dSt||k rh|S|jr|rtj||j|jjd\}|_|jjdkrt|ddst|}|jjrt}||||}tt|d|_|s|S|jdk rz|jtjkrt ||j}nd|jj!p"|jj"}tj#|jjk}|jtj$k}tj%|jjkrVt&|}t'||||d(|jj)|_ }n"|j*dk r|jj+d rz,d d l,m-d d l,m.d d l/m0} m1} Wnt2k rt2dYnXd} d} t|dd} |jj+3d| }|jj+dd kr.t4j5| |jj+dd} nR|dk rt6}|D].}t7||}|dkrdt8dn |9|qBt4j5| |d} |jj:}|jj+drfdd} |f| | d||_ }difdd}|t;_j?|t@tABdgd}n|jjCtDjEkrtFr|Si}|jjGdk r8|jjG|d<n"t|tHrR|jI |d<nd|d<|jjJdk rt|jjJ|d<|jjKdk r|jjK|d <tLf||jM_N|S)!N)r)Zbackward_passes_per_step) opt_levelr Zis_loaded_in_8bitF)mixed_precisionZreshard_after_forward cpu_offloadrr)XlaFullyShardedDataParallel)checkpoint_module)size_based_auto_wrap_policytransformer_auto_wrap_policyzJMissing XLA FSDP related module; please make sure to use torch-xla >= 2.0.Z_no_split_modulesZtransformer_layer_cls_to_wrapmin_num_params)rz@Could not find the transformer layer class to wrap in the model.)Ztransformer_layer_clsZxla_fsdp_grad_ckptcs|f||SNr)mrkwargsZFSDPrrrauto_wrapper_callablesz2Trainer._wrap_model..auto_wrapper_callable)auto_wrap_policyrcSs|jf|}|rt|Sr)rrV mark_step)rbarrierZoptimizer_argslossrrrpatched_optimizer_steps z3Trainer._wrap_model..patched_optimizer_stepZSMDATAPARALLEL_LOCAL_RANK)Z device_idsZfind_unused_parametersTZ bucket_cap_mbZbroadcast_buffers)OrZuse_ipexrrrrrrxrrrrZDistributedModelrDr)rrZ initializerZfp16_opt_leveln_gpurrZ DataParallelZ jit_mode_evaltimerroundjit_compilation_timerrTr ShardedDDPrrZOFFLOADrZ AUTO_WRAPrrr%rrrZtorch_xla.distributed.fsdprrZtorch_xla.distributed.fsdp.wraprrrr functoolspartialrrB ExceptionaddZxla_fsdp_configrVZoptimizer_steprwparallelZDistributedDataParallelrrgetenvrrbrrzZddp_find_unused_parametersr'Zis_gradient_checkpointingZddp_bucket_cap_mbZddp_broadcast_buffersrrOZ ddp_handler)rrrFrr start_timerrZzero_3rrr rZ%default_transformer_cls_names_to_wrapZ"fsdp_transformer_layer_cls_to_wrapZtransformer_cls_to_wrapZ layer_classZtransformer_clsZ fsdp_kwargsr rrrr _wrap_modelMs                 zTrainer._wrap_model)resume_from_checkpointrignore_keys_for_evalcKs|dkr d}|j|j}d|_|js.|jrD|jsD||j|j d|krb| d}t dt t|dkrtddt|d |||jj|_d}|jdk r|jjrt|jjn t|jj|||_d}d \|_|_t|tr"|r"t |j!}|dkr"t"d |j!d |dk rNt#sN|j$sN|j%sN|&||rt|j'rl||j|j |j|_(t)|j*|j|j+}|j,rzt-/|||||d WSt-.Xn|||||d SdS)a Main training entry point. Args: resume_from_checkpoint (`str` or `bool`, *optional*): If a `str`, local path to a saved checkpoint as saved by a previous instance of [`Trainer`]. If a `bool` and equals `True`, load the last checkpoint in *args.output_dir* as saved by a previous instance of [`Trainer`]. If present, training will resume from the model/optimizer/scheduler states loaded here. trial (`optuna.Trial` or `Dict[str, Any]`, *optional*): The trial run or the hyperparameter dictionary for hyperparameter search. ignore_keys_for_eval (`List[str]`, *optional*) A list of keys in the output of your model (if it is a dictionary) that should be ignored when gathering predictions for evaluation during the training. kwargs (`Dict[str, Any]`, *optional*): Additional keyword arguments used to hide deprecated arguments FNTZ model_pathzi`model_path` is deprecated and will be removed in a future version. Use `resume_from_checkpoint` instead.rz3train() received got unexpected keyword arguments: r1rrz/No valid checkpoint found in output directory ())rrrr)0rrrrrrrrrrrrrrrrr:r+r,rrrrrrYrr_rrrrboolr[rrrxrr_load_from_checkpointrrrZ_inner_training_loopZauto_find_batch_sizer hf_hub_utilsZenable_progress_barsZdisable_progress_bars)rrrrrrZmodel_reloadedZinner_training_looprrrrsz              z Trainer.trainc/ Cs |j||_td|j|}|j|j|j}d}d} t|r t |}||j} t | d} | |} |j dkr|j } |j | t |j | dk} |j |}|jr|||j |j} nDt|j| } t|j} | ||j}|jrj|||j} n^|j dkrZ|j } tj} | } ||j } |j |}|jrj|||j |j} ntd|j tj|jjkr|jjdkrtdn t|j}|jdk r|jtjkptp|j dk p|j!}|j"rd|_#d|_"|j$rt%|| d\|_&|_#|s|j'| dt(|_)|dk |j)_*|j+dk r^|j+dkrTt| |j+|j)_+n |j+|j)_+|j,dk r|j,dkrt| |j,|j)_,n |j,|j)_,|j-dk r|j-dkrt| |j-|j)_-n |j-|j)_-|j.r|j/|0|j1}||jkrdnd}|r(|r|j2|j|_|j'| d|r|j3t4|j#d rx|j5r^|j2|j}n|j2|j|j&\}|_&n |j2|j|j&|j#\}|_&|_#|j!r||_|_1||jk r||_1|j$r|j1|_6|dk r |j$rt7|j1|nts|j!r |8||j1|9|t:d t:d | d t:d | d t:d|jj;d |jj;|jkr~t:d|jd t:d|d t:d|jt:d| d t:dt<|ddd d|j)_=t>>}d}d}d}|dk rt?j@At?j@B|tCrt(Dt?j@B|tC|_)|j)jE| }|jFsN|j)jE| }||j9}nd}t:dt:d|t:d|j)jE|jFst:d|d|d|j|jG_|j&|jG_&|j#|jG_#||jG_H|jIdk r|jJdk r|I|jJ|j)_K|dk r|jLtMjNkr |jOn|}tP||j)_Qnd|j)_Q| |j)_ | |j)_|R|j)_R|S|j)_StTUdV|jW}d|_X|j)jE|_Y|Z|jG[||j)|j\|_\|jFst]|D]R}t^|}t_|t`}tas|s|D] }qqn|dk r|ng}tb|}qd}t]|| D]}|}|jcdkrd|_d|dk r.t |n |j |j}|jGe||j)|j\|_\||krx|dk rx|dkrx|f|d} d}!|dkrtg||}|}!d}d} d}"th|D]\}"}#|d7}| r|f|d} |dkr|d8}|dk r|id|dkr|f|qn|dk r(|jd}|"|jdkrN|jGk||j)|j\|_\|jl||m||#}$W5QRX|jnrtostTp|$stTq|$r||d|j)jE|jY7}n||$7}|jrts|t|#7_r||jko|"d|k}%||jdk s|% r |% s tuvtwtuvdk r.|jjxyd|jzdk r|jzdk r|j{ rto rzt|}|j&}&t|j~d|&dt|d |j|j&t r|j r|j&|jznnt4|j&d! r|j&|jznPt4|d" r||jzn6|j5 rtjt|j&|jzn|j||jzd}'to rP|j{ rD|j|j&|jin |j&nR|j{ r|j}(|j|j&|ji|j})|(|)k}'n|j&|jj }'|' rt_|j#tTjj#j s|j#|Z|j)jEd7_E||"d|!||j)_=|jG||j)|j\|_\||||||n|jG||j)|j\|_\|j\j sJ|j\jr qTq|"dk rtd#|j)jEd$| d%d|j\_|jG||j)|j\|_\||||||tj|jjk rto rt|tn td&|j\jr qq|jc rt4|d' rt|d't:d(|j rr|j)jdk rrto rBt|d)n(|jtjk rZtnt rjt||jX|7_X|jX|j)jE}*td*|||j)j | d+}+||j)j|+d,<|*|+d-<d|_|j|+||+||},|jd|,d.}-|jj rZ|j)jdk rZ|jjdk rZ|-D]6}.t?j@|.|j)j s"t:d/|.d0t|. q"|jG||j)|j\|_\|t|j)jE|*|+S)1Nz)Currently training with a batch size of: r rzYargs.max_steps must be set to a positive value if dataloader does not have a length, was zjCurrently --debug underflow_overflow is not supported under DP. Please use DDP (torch.distributed.launch).Fr_Trz***** Running training ***** Num examples = rz Num Epochs = z( Instantaneous batch size per device = zA Training with DataParallel so batch size has been adjusted to: zE Total train batch size (w. parallel, distributed & accumulation) = z Gradient Accumulation steps = z Total optimization steps = z# Number of trainable parameters = )Ztrainable_onlyzE Continuing training from checkpoint, will skip to saved global_stepz! Continuing training from epoch z' Continuing training from global step z Will skip the first z epochs then the first z batches in the first epoch.rprrzg?)scaleclip_grad_normclip_grad_norm_zXThere seems to be not a single sample in your epoch_iterator, stopping training at step zI! This is expected if you're using an IterableDataset and set num_steps (z.) higher than the number of available samples.zYou enabled PyTorch/XLA debug metrics but you don't have a TPU configured. Check your training configuration if this is unexpected._pastzU Training completed. Do not forget to share your model on huggingface.co/models =) load_best_model_at_endr) num_samples num_stepsr total_flos train_loss use_mtimerDeleting older checkpoint [] due to args.save_total_limit)rOZ free_memoryrrr{rRrDrXr\rmaxrrrZinclude_tokens_per_secondrmathceilnum_train_epochssysmaxsizerrZUNDERFLOW_OVERFLOWrrrrrrTrrxrrrrrr#rrer7r Zis_hyper_param_searchZ logging_stepsZ eval_stepsZ save_stepsgradient_checkpointingZgradient_checkpointing_enablerrrPrrrrr$r_load_optimizer_and_schedulerrrrAepochrrrisfiler:rZload_from_jsonrZignore_data_skiprtrain_dataloaderrrZ trial_namerrOrrrZ trial_paramsrrrZtensorr%r_total_loss_scalar_globalstep_last_loggedZ zero_gradZon_train_beginrranger@rrr/r+ past_indexr'Zon_epoch_begin_load_rng_staterrrcloseZ on_step_begin accumulate training_stepZlogging_nan_inf_filterr{isnanisinfrfloatfloating_point_opsrraccelerate_versionZgradient_stateZ_set_sync_gradientsrr rVZ_fetch_gradientsZ all_reducerWrZunscale_rZclip_master_gradsr%r&rrrZ master_paramsrrZ get_scaleZoptimizer_step_was_skippedrqReduceLROnPlateauZ on_step_end_maybe_log_save_evaluateZon_substep_endZshould_epoch_stopZshould_training_stopr Z on_epoch_endTPU_METRICS_DEBUG master_printmetmetrics_reportdelattrr(best_model_checkpoint rendezvousrrbrdistr r_load_best_modelitemr` store_flosr+rrrlog_get_output_dir_sorted_checkpointsrsave_total_limitsamefileshutilrmtreer_finish_current_pushrV)/rrIrrrrr;Ztotal_train_batch_sizeZlen_dataloaderZnum_train_tokensZnum_update_steps_per_epochrrr4Znum_train_samplesZdebug_overflowZdelay_optimizer_creationrZuse_accelerator_preparerZepochs_trainedZsteps_trained_in_current_epochZsteps_trained_progress_barrtr_lossr9rJZis_random_sampler_Ztotal_batched_samplesZepoch_iteratorZsteps_in_epochZ rng_to_syncZ steps_skippedrinputsZ tr_loss_stepZ)is_last_step_and_steps_less_than_grad_accZ gradientsZoptimizer_was_runZ scale_beforeZ scale_afterr,rrun_dircheckpoints_sorted checkpointrrrr >s                                                                                             &zTrainer._inner_training_loopcCs|jdk r|dk r|jtjkr&|j}nR|jtjkrHddlm}|}n0|jtjkr\|j }n|jtj krxddl }|j j }|j dk r| |nd|}tj|jj|}n|jj}|S)Nrrzrun-)rrOrnumberrrrZ get_trial_idridrrrunrrrr:rr)rrZrun_idrrZrun_namerarrrrWs      zTrainer._get_output_dircs|dkr|j}tjt}tjt}tjt}tjt}tjt}tjt }tjt } tj ot fddt D} | r|jstddt dd|||| ||fDs| stdtddtj|r>t|} | j} | dk r>| tkr>td | d td tj|s`tj|s`| rTtrtjtjd rtjtd d dnLt|jdr|jjdkrtdtj|dd} d | d<|j | dd}~ np|jrt!|j"j#j$|j"|nP|jj%r,tj|r,t&jj'|dd} ntj|dd} | | d }~ |(|nt)rt*|t+rt|drt|drtj,r|j-|j.ddntdtdn tdn(t/|t|jj%d}ts|(|dS)Nc3s6|].}tjtj|rtdd|kVqdSrrN)rrisdirr:rkrrZ folder_namerrr 2sz0Trainer._load_from_checkpoint..zCheckpoint found at z* is only supported when using PyTorch FSDPcss|]}tj|VqdSr)rrr:)rfrrrrk<sz!Can't find a valid checkpoint at zLoading model from rz9You are resuming training from a checkpoint trained with z- of Transformers but your current version is zJ. This is not recommended and could yield to errors or unwanted behaviors.user_content.ptFrtagrZload_optimizerrTzPEnabling FP16 and loading from smp < 1.10 checkpoint together is not suppported.r map_location_smp_is_partialrractive_adapter load_adapter)Z is_trainablejThe intermediate checkpoints of PEFT may not be saved correctly, consider using a custom callback to save i in corresponding saving folders. Check some examples here: https://github.com/huggingface/peft/issues/96GCould not load adapter model, make sure to have `peft>=0.3.0` installed)rZ prefer_safe)0rrrr:rgrfrerkrjrirhrhanylistdirrrrrr:rZfrom_json_fileZtransformers_versionrr rxrrrrrrloadload_state_dictrrOr  fsdp_pluginsave_safetensors safetensors load_file_issue_warnings_after_loadrurrexistsrurtr()rrr config_fileZadapter_weights_fileZadapter_safe_weights_fileZ weights_fileZweights_index_fileZsafe_weights_fileZsafe_weights_index_fileZ is_fsdp_ckptconfigZcheckpoint_versionr load_resultrrjrr's   "    zTrainer._load_from_checkpointc Cstd|jjd|jjdtj|jjt}tj|jjt }tj|jjt }tj|jjt }t rt|j n|j}|jrt|j |jjn4|jrt|jjj|j||jj}ntj|stj|stj|stj|rrd}t rztjtj|jjdr*tj|jjtdddnN|jjrTtj|rTtjj|dd }ntj|dd }d|d <|j|dd }nt rt!|t"rt#|d rt#|drtj|stj|r|$|jj|j%ddl&m'} | gg}nt(dt dd}nt(dd}nD|jjr>tj|r>tjj|dd }ntj|dd }||d}t s|r|)|nVtjtj|jjt*rt+||jjt d }t s|)|nt(d|ddS)NzLoading best model from z (score: z).TrmFrnrrsrprrrrtrur)_IncompatibleKeysrvrwrxz#Could not locate the best model at zi, if you are running a distributed training on multiple nodes, you should activate `--save_on_each_node`.),rrr rP best_metricrrr:rkrirfrerxrrrr$rrrOr}rr:rrrr~rrrr{r|rurrrrurtZtorch.nn.modules.modulerr rrjr() rZbest_model_pathZbest_safe_model_pathZbest_adapter_model_pathZbest_safe_adapter_model_pathrrZhas_been_loadedrrrrrrSs              zTrainer._load_best_modelcCsvt|jdkrP|jjdk r       zTrainer._load_rng_statec Cstd|jj}|jdkr,|dkr,||j|d}tj||}|j |dd|j rf|j ||j tjkr||j|js|jr|jrt|jjj|j|j|j|n*|jj|j|j}t|tj|ttrBtdt|j tj|tt!j"dd*}t|j# tj|t$t%|W5QRXnt&r|jj'dd} t()t(*d kszt(jj+j,rt(j| tj|tdt(jj+j,d n>|j-j.r|j s|js|jst|j tj|t|j ot/|j#t0 } |j-j.rr|j r | rrtsrt!j"dd"}t|j# tj|t$W5QRXt%||j1rrt|j2 tj|t3|dk r|j-j4dk r|j-j4} | 5d sd | } || } |j-j6rt7j8nt7j9} |jj:dks|jj;dks| | |jj:r| |j_:||j_;|j-j.r"|j?t7j>@tj>Ad }tjBCr||j-jDtEjFkrltjBj>G|d <ntjBj>A|d <trtA|d<tjH|dd|j-jIdkrt|tj|dn"t|tj|d|j-jJd|j-jKr|L||j-j.r|jMd|ddS)NrrTrZsaving_optimizer_statesrecordF)Zgather_if_shardr)rZv3r)rrrrrrr rrrr-)NrJr rrrUrWrrr:rrrsave_checkpointrrTrrZconsolidate_state_dictrrrrOr}rrZfull_optim_state_dictrrrr{rVrQrrcatch_warningsrrrHrxZlocal_state_dictrr Zrdp_rankr Zshard_optimizer_staterrrrr r SCALER_NAMErrZgreater_is_betterrZgreaterlessrrPrrrgetstateZ get_stateZ get_rng_staterrrrbrZget_rng_state_allrrXrr_push_from_checkpoint_rotate_checkpoints)rrrrcheckpoint_folderrarfull_osdcaught_warningsZopt_state_dictZis_deepspeed_custom_schedulerrZ metric_valueoperatorZ rng_statesrrrrO s       " &        "   zTrainer._save_checkpointc sdkr dS|jr^t|jtsZtjdd"}|jtt j t W5QRXt |dStr|tt j tdnRt j t j tpt j t j tpt j otfddt D}|rt j t j t rtrtjt j tdd}tjdd}tjt j t dd}W5QRXt |t||jjt||jj|j||j|nvtrt j t j d rfd d }n fd d }|j|n|jjd kr|jjnd}|js|j r`|j rt!|j"j#j$|j"|j|j%nDd}|jj&dkr@tt j t}|j%j'(||j%} |j| n |jtjt j t|dtjdd"}|jtt j t W5QRXt ||j)rt j t j t*r|j+tt j t*dS)z3If optimizer and scheduler states exist, load them.NTr_*c3s6|].}tjtj|rtdd|kVqdSrg)rrrhr:OPTIMIZER_NAME_BINrrircrrrk sz8Trainer._load_optimizer_and_scheduler..rrprmcs"|tjtjtdddS)NTr)r|rr{rrr:rmodoptrrr opt_load_hook sz.opt_load_hookcsHtr&|tjtjtdddn|tjtjtdddS)NT)rZ back_compatr)rr|rr{rrr:rrrrrr s r r),rrrrrrr|rr{rrr:rrHrxglobrr:rrhryrzr{rVZsend_cpu_data_to_devicerrrrZregister_post_step_hookrXrrrrOr r}rrrZscatter_full_optim_state_dictr rr) rrcrZcheckpoint_file_existsZoptimizer_stateZlr_scheduler_staterrqrZ sharded_osdrrrr8 sp &  "  & z%Trainer._load_optimizer_and_schedulerminimizer)rrn_trials directionbackendrr?c Ks|dkrt}t|}t|}|||_|jdkr@td|dkrN|jn||_||_ |dkrft n||_ |j |||f|} d|_| S)aD Launch an hyperparameter search using `optuna` or `Ray Tune` or `SigOpt`. The optimized quantity is determined by `compute_objective`, which defaults to a function returning the evaluation loss when no metric is provided, the sum of all metrics otherwise. To use this method, you need to have provided a `model_init` when initializing your [`Trainer`]: we need to reinitialize the model at each new run. This is incompatible with the `optimizers` argument, so you need to subclass [`Trainer`] and override the method [`~Trainer.create_optimizer_and_scheduler`] for custom optimizer/scheduler. Args: hp_space (`Callable[["optuna.Trial"], Dict[str, float]]`, *optional*): A function that defines the hyperparameter search space. Will default to [`~trainer_utils.default_hp_space_optuna`] or [`~trainer_utils.default_hp_space_ray`] or [`~trainer_utils.default_hp_space_sigopt`] depending on your backend. compute_objective (`Callable[[Dict[str, float]], float]`, *optional*): A function computing the objective to minimize or maximize from the metrics returned by the `evaluate` method. Will default to [`~trainer_utils.default_compute_objective`]. n_trials (`int`, *optional*, defaults to 100): The number of trial runs to test. direction (`str` or `List[str]`, *optional*, defaults to `"minimize"`): If it's single objective optimization, direction is `str`, can be `"minimize"` or `"maximize"`, you should pick `"minimize"` when optimizing the validation loss, `"maximize"` when optimizing one or several metrics. If it's multi objectives optimization, direction is `List[str]`, can be List of `"minimize"` and `"maximize"`, you should pick `"minimize"` when optimizing the validation loss, `"maximize"` when optimizing one or several metrics. backend (`str` or [`~training_utils.HPSearchBackend`], *optional*): The backend to use for hyperparameter search. Will default to optuna or Ray Tune or SigOpt, depending on which one is installed. If all are installed, will default to optuna. hp_name (`Callable[["optuna.Trial"], str]]`, *optional*): A function that defines the trial/run name. Will default to None. kwargs (`Dict[str, Any]`, *optional*): Additional keyword arguments passed along to `optuna.create_study` or `ray.tune.run`. For more information see: - the documentation of [optuna.create_study](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.study.create_study.html) - the documentation of [tune.run](https://docs.ray.io/en/latest/tune/api_docs/execution.html#tune-run) - the documentation of [sigopt](https://app.sigopt.com/docs/endpoints/experiments/create) Returns: [`trainer_utils.BestRun` or `List[trainer_utils.BestRun]`]: All the information about the best run or best runs for multi-objective optimization. Experiment summary can be found in `run_summary` attribute for Ray backend. NzXTo use hyperparameter search, you need to pass your model through a model_init function.) r"rOr!Zensure_availablerrrZdefault_hp_spacerrrWrrf) rrrrrrrrZ backend_objZbest_runrrrhyperparameter_search& s ;  zTrainer.hyperparameter_search)rr?cCsZ|jjdk rt|jjd|d<|d|jji}|jj||j|j|j|j ||_ dS)z Log `logs` on the various objects watching training. Subclass and override this method to inject custom behavior. Args: logs (`Dict[str, float]`): The values to log. Nr9r) r r9rrZ log_historyrrZon_logrr)rroutputrrrrVu s z Trainer.log)rr?cst|tr(t|fdd|DSt|ttfrPt|fdd|DSt|tjrdjj i}j rt |st |r| djjjji|jf|S|S)z| Prepares one `data` before feeding it to the model, be it a tensor or a nested list/dictionary of tensors. csi|]\}}||qSr_prepare_inputrrrrrs sz*Trainer._prepare_input..c3s|]}|VqdSrr)rrrrrrk sz)Trainer._prepare_input..rr)rrr6rrr+rTensorrrrZis_floating_pointZ is_complexrrOr rrrr%)rrrrrrr s    zTrainer._prepare_input)r`r?cCsR||}t|dkr.tdd|jd|jjdkrN|jdk rN|j|d<|S)z Prepare `inputs` before feeding them to the model, converting them to tensors if they are not already and handling potential state. rzThe batch received was empty, your model won't be able to train on it. Double-check that your training dataset contains keys expected by the model: rrNZmems)rrrr:rrr?r'rr`rrrr s   zTrainer._prepare_inputscCs|S)zF A helper wrapper to group together context managers. )autocast_smart_context_managerrrrrcompute_loss_context_manager sz$Trainer.compute_loss_context_managerrcCsH|js |jr<|jr&tjjj||jdntjjj||jd}nt }|S)z A helper wrapper that creates an appropriate context manager for `autocast` while feeding it the desired arguments, depending on the situation. )rr) rrrrrrrr contextlib nullcontext)rrZ ctx_managerrrrr s z&Trainer.autocast_smart_context_manager)rr`r?c Cs|||}tr>t|||jj}||jj S| | ||}W5QRX|jj dkrr| }|jr|j|n6|jrt||j}|W5QRXn |j|||jjS)aq Perform a training step on a batch of inputs. Subclass and override to inject custom behavior. Args: model (`nn.Module`): The model to train. inputs (`Dict[str, Union[torch.Tensor, Any]]`): The inputs and targets of the model. The dictionary will be unpacked before being fed to the model. Most models expect the targets under the argument `labels`. Check your model's documentation for all accepted arguments. Return: `torch.Tensor`: The tensor with training loss on this batch. r )rrrxrrrD reduce_meandetachr%rr compute_lossrrr rr$ZbackwardrrZ scale_lossrrO)rrr`loss_mbr Z scaled_lossrrrrC s     zTrainer.training_stepcCs|jdk rd|kr|d}nd}|f|}|jjdkrF||jj|_|dk rtrnt|trnt|j  }n t| }|t kr|j||dd}q|||}nVt|t rd|krtdd|d d|d t|t r|dn|d}|r ||fS|S) z How the loss is computed by Trainer. By default, all models return the loss in the first element. Subclass and override for custom behavior. NlabelsrT)Z shift_labelsr zJThe model did not return a loss from the inputs, only the following keys: rz,. For reference, the inputs it received are r)rrrr?r'rurrr)Z base_modelZ _get_namer*rrrr:r,)rrr`return_outputsroutputsr@r rrrr s&     $zTrainer.compute_losscCs |jjdkS)z Whether or not this process is the local (e.g., on one machine if training in a distributed fashion on several machines) main process. r)rZlocal_process_indexrrrrr szTrainer.is_local_process_zerocCs"trtdkS|jjdkSdS)z Whether or not this process is the global main process (when training in a distributed fashion on several machines, this is only going to be `True` for one process). rN)rxrrTrrrrrrr s zTrainer.is_world_process_zero)rrcCs|dkr|jj}tr$||ntrvtj|dd|j}|jj rX|j ||dt rrt tj |dnNtj|jjkstj|jjks|jdk s|jr|js|jni}|jj r|j ||d|jrt|jj |ttgt|jjj|j|j|n|jrt t!t dkr*t"dz*|j#|j$}|jj rR|j ||dWnXt"k rt%&d|jj r|j |idt|jj |ttg|j'|YnXn|jj r| ||jj(r|s|j(d d dS) z Will save the model, so you can reload it using `from_pretrained()`. Will only save from the main process. NTr)rrmrz#Install Accelerate from main branchz| stage3_gather_16bit_weights_on_model_save=false. Saving the full checkpoint instead, use zero_to_fp32.py to recover weightsz Model save)commit_message))rrr{ _save_tpurxrrrrr_saverrrr:touchrTrrrrrrrIrkrirrOr r}rrrrHrZget_state_dictrrr rr)rrrrrrrr sX       zTrainer.save_modelrcCs|dk r |n|jj}td|trRtj|ddt |jtj |t t dt|jtstt|jtrt|jj||jj|jtj dqtd|j}t |tj |tn|jj||jjtj d|jdk r|jjr|j|dS)NSaving model checkpoint to TrZsaving_checkpoint)is_main_processr save_functionETrainer.model is not a `PreTrainedModel`, only saving its state dict.)rr)rrrrrVZis_master_ordinalrrrrrr:TRAINING_ARGS_NAMErQrrr'r)save_pretrainedrrrkr)rrrrrrr[ s(     zTrainer._save_tpucCs|dk r |n|jj}tj|ddtd|ts>tfnttf}t |j |s|dkrd|j }t t |j |rt |j j |||jjdqtd|jjrtj|tj|tqt|tj|tn|j j |||jjd|jdk r|j |t|jtj|tdS)NTrr)rZsafe_serializationr)rrrrrrrur'rrrrr)rr~rrZ save_filerr:rirrkrr)rrrZsupported_classesrrrrw s2      z Trainer._savecCsZ|jjtjkr>|jjt|jg|jjd 7_d|_n|jj|j7_d|_dS)Nrsr) rrrbrr r+r=rrrzrTrrrrrU szTrainer.store_flosc Csg}ddt||dD}|D]b}|rF|tj||fq&td|d|}|dk r&|dk r&|t |d|fq&t |}dd|D}|j j dk r| tt|j j } t| t|dD]&} || d || || <|| d <q|S) NcSs g|]}tj|rt|qSr)rrrhr)rxrrrr s z/Trainer._sorted_checkpoints..z-*z.*z -([0-9]+)rcSsg|] }|dqS)r r)rrcrrrr srr )rrrrrgetmtimerematchgroupsrsortedr rPindexrr>r) rrZcheckpoint_prefixr.Zordering_and_checkpoint_pathZglob_checkpointsrZ regex_matchrbZbest_model_indexirrrrX s $zTrainer._sorted_checkpointscCs|jjdks|jjdkrdS|j||d}t||jjkr>dS|jj}|jjdk rr|jjdkrr|d|jjkrrd}tdt||}|d|}|D]$}td|dt j |dd qdS) Nrr-r r#rr/r0T) ignore_errors) rrYrXrr rPr1rrr[r\)rr.rrbrYZnumber_of_checkpoints_to_deleteZcheckpoints_to_be_deletedrcrrrr s$   zTrainer._rotate_checkpointsr)rrrr?c Cs|j||}t}|jjr*|jn|j}||d|jdkrDdnd||d}|jj |jj }|d|j kr||j |d7}|j t |||jt|j|d||j tj|jjkrtt|j|j|j|j|j |_|j|j |j S)a' Run evaluation and returns metrics. The calling script will be responsible for providing a method to compute metrics, as they are task-dependent (pass it to the init `compute_metrics` argument). You can also subclass and override this method to inject custom behavior. Args: eval_dataset (`Dataset`, *optional*): Pass a dataset if you wish to override `self.eval_dataset`. If it is a [`~datasets.Dataset`], columns not accepted by the `model.forward()` method are automatically removed. It must implement the `__len__` method. ignore_keys (`List[str]`, *optional*): A list of keys in the output of your model (if it is a dictionary) that should be ignored when gathering predictions. metric_key_prefix (`str`, *optional*, defaults to `"eval"`): An optional prefix to be used as the metrics key prefix. For example the metrics "bleu" will be named "eval_bleu" if the prefix is "eval" (default) Returns: A dictionary containing the evaluation loss and the potential metrics computed from the predictions. The dictionary also contains the epoch number which comes from the training state. Z EvaluationNT)r/prediction_loss_onlyrr_jit_compilation_timer)r*)rrr[rrrUprediction_loopevaluation_looprrZrXrrr`r)r2r3rVrrKr{rVrLrMrNrZ on_evaluater rr) rrrreval_dataloaderr eval_looprtotal_batch_sizerrrr s8    zTrainer.evaluater])r\rrr?c Cs|j||}t}|jjr*|jn|j}||d||d}|jj|jj }|d|j krt||j |d7}|j t |||j t|j |d|j|j|j|j|j |_|j|j t|j|j|j dS)a Run prediction and returns predictions and potential metrics. Depending on the dataset and your use case, your test dataset may contain labels. In that case, this method will also return metrics, like in `evaluate()`. Args: test_dataset (`Dataset`): Dataset to run the predictions on. If it is an `datasets.Dataset`, columns not accepted by the `model.forward()` method are automatically removed. Has to implement the method `__len__` ignore_keys (`List[str]`, *optional*): A list of keys in the output of your model (if it is a dictionary) that should be ignored when gathering predictions. metric_key_prefix (`str`, *optional*, defaults to `"test"`): An optional prefix to be used as the metrics key prefix. For example the metrics "bleu" will be named "test_bleu" if the prefix is "test" (default) If your predictions or labels have different sequence length (for instance because you're doing dynamic padding in a token classification task) the predictions will be padded (on the right) to allow for concatenation into one array. The padding index is -100. Returns: *NamedTuple* A namedtuple with the following keys: - predictions (`np.ndarray`): The predictions on `test_dataset`. - label_ids (`np.ndarray`, *optional*): The labels (if the dataset contained some). - metrics (`Dict[str, float]`, *optional*): The potential dictionary of metrics (if the dataset contained labels). Z Prediction)r/rrrr) predictionsr'r)rrr^rrrUrrrZrXrrr`r)r2r3rZ on_predictr rrrRrr') rr\rrZtest_dataloaderrrrrrrrpredict s0$   zTrainer.predict)rr/rrrr?c Cs|j}|dk r|n|j}|jr:|jdkr:t|ddd\}}|j|jd|d}t|jj dkr||jkr|jrx|j |n|jj |dd}|j r||_||jk r||_ |jr|j |_|js|jr|jtj|jd}n|jr|jtj|jd}|jj} td |d t|r&td ||n td td | |||j_t|dd} |jdkrnd|_ d} d} d} d}d}d}d}d}d}t!|D]\}}t"|}|dk r||7}| dkr|} |j#||||d\}}}t|jdd}|j$r|%||nd}t&rt'(|dk rT|j)|*| }| dkrF|n t+| |dd} |dk rp|jj,|ddd}|dk r|jj,|ddd}|j)|}|dkr|n t+||dd}|dk r|jj,|ddd}|j-dk r|-||}|j)|}| dkr|n t+| |dd} |dk rB|j)|}| dkr4|n t+| |dd} |j.||j/|j0|_0|j1dk r|d|j1dkr|jj2st34t5t34dkr| dk rt6| }|dkr|nt7j8||fdd}| dk rt6| }|dkr|n t+||dd}|dk r&t6|}|dkr|n t+||dd}| dk rTt6| }|dkrF|n t+||dd}d\} } }} q|jrt9|drt:|d| dk rt6| }|dkr|nt7j8||fdd}| dk rt6| }|dkr|n t+||dd}|dk rt6|}|dkr|n t+||dd}| dk r>t6| }|dkr0|n t+||dd}t| rRt| }n@t;| t<rxt| dddkrx| j}nt|r||}n|}|dkr|dkr|}|j=dk r|dk r|dk r|j$r|=t>|||d}n|=t>||d}ni}t?|}|dk r(|@A||d<t9|drD|jB||d<tC|DD]0}|E|d sP|F|||d |<qPtG||||d!S)" Prediction/evaluation loop, shared by `Trainer.evaluate()` and `Trainer.predict()`. Works both with or without labels. NrTr`Z inferenceFrFrZevaluation_moderr***** Running  *****r"z Num examples: Unknown Batch size = r.rmain_input_namerZ padding_indexr )dimZ pad_indexr)ZaxisNNNNr'rrr'r`rr'_lossrrr_rr'rr))Hrrrrr#rrrrO_modelsrP prepare_modelrrrrr%rr rrrrZrrr\rrrrrr?r'rr?prediction_stepinclude_inputs_for_metricsrr{rVr Zgather_for_metricsrepeatrDZpad_across_processesron_prediction_stepr reval_accumulation_stepsZsync_gradientsrrrHrFrZ concatenaterrOrr9rrMrXrrTrr+r,rrrL) rrr/rrrrr_rrIr losses_host preds_host labels_host inputs_hostZ all_lossesZ all_predsZ all_labelsZ all_inputsZobserved_num_examplesrr`Zobserved_batch_sizer logitsrr inputs_decodelossesr)rrrrrrZ s                                       zTrainer.evaluation_loopcCsx|dkr dStr*|dkrd}t||}nJtr:t|}n:|jjdk rT|jjjdksl|jjdkrt|jjdkrtt|}|S) Gather value of `tensors` (tensor or list/tuple of nested tensors) and convert them to numpy before concatenating them to `gathered` NZ nested_gatherNOr#) r{rGrxrrZdistributed_stateZdistributed_typeZ local_rankr>rZtensorsrgrrrr2 s    zTrainer._nested_gather)rr`rrr?c st|jdkrdntfdd|jD}dd}|dkrD|j}t|jdkrZ|rZdnd}|dkrt|jdrt|jj d gng|s|rt t fd d|jD}t|d kr|d}nd}t trt|} |s|rVt| tr(| d } t fd d| D} n| d} | d d} | } t| } n8d} t| trt fdd| D} n| } t| } n|s|r||j|dd\} }W5QRX| } t|trt fdd|D} n |d d} nnd} ||f}W5QRXt|trPt fdd|D} n|} |jjdkrt||jjd |_W5QRX|r| ddfSt | } t| d kr| d} | | |fS)a Perform an evaluation step on `model` using `inputs`. Subclass and override to inject custom behavior. Args: model (`nn.Module`): The model to evaluate. inputs (`Dict[str, Union[torch.Tensor, Any]]`): The inputs and targets of the model. The dictionary will be unpacked before being fed to the model. Most models expect the targets under the argument `labels`. Check your model's documentation for all accepted arguments. prediction_loss_only (`bool`): Whether or not to return the loss only. ignore_keys (`List[str]`, *optional*): A list of keys in the output of your model (if it is a dictionary) that should be ignored when gathering predictions. Return: Tuple[Optional[torch.Tensor], Optional[torch.Tensor], Optional[torch.Tensor]]: A tuple with the loss, logits and labels (each being optional). rFc3s|]}|dk VqdSrrr3r`rrrkc sz*Trainer.prediction_step.. return_lossNTrZkeys_to_ignore_at_inferencec3s|]}|VqdSrr rfr rrrku sr r c3s$|]\}}|dgkr|VqdSr Nrrrrrrk sc3s|]\}}|kr|VqdSrrrrrrrk s)rc3s$|]\}}|dgkr|VqdSrrrrrrrk sc3s|]\}}|kr|VqdSrrrrrrrk s)rrallrrmrrrrrrErrrrxrrrrrrrrrrrrr?r')rrr`rrZ has_labelsrZloss_without_labelsrZ raw_outputsrZ logits_mbr rrr)rr`rrE sh*                    zTrainer.prediction_stepr cCs t|jdr|j|SdSdS)a For models that inherit from [`PreTrainedModel`], uses that method to compute the number of floating point operations for every backward + forward pass. If using another model, either implement such a method in the model or subclass and override this method. Args: inputs (`Dict[str, Union[torch.Tensor, Any]]`): The inputs and targets of the model. Returns: `int`: The number of floating-point operations. rGrN)rrrGrrrrrG s  zTrainer.floating_point_opscCs^|s dS|jjdkr,t|jjj}n|jj}t||jj|jj dd}|j |_d|_ dS)zE Initializes a git repo in `self.args.hub_model_id`. NT)tokenprivater) rrrrrabsolutergr hub_tokenhub_private_reporepo_idpush_in_progress)r repo_namerepo_urlrrrr s zTrainer.init_hf_repo)at_initc CsDtd|sdS|jj}|dkr8t|jjj}t ||jj |jj ddj }zt |jj||jj d|_WnJtk r|jjr|rt|jjt |jj||jj d|_nYnX|jtjtj|jjds$|jjtjkr$ttj|jjdddd }|d gW5QRXtjd r:|d|_ dS) a Initializes a git repo in `self.args.hub_model_id`. This function is deprecated and will be removed in v4.34.0 of Transformers. Args: at_init (`bool`, *optional*, defaults to `False`): Whether this function is called before any training or not. If `self.args.overwrite_output_dir` is `True` and `at_init` is `True`, the path to the repo (which is `self.args.output_dir`) might be wiped out. zy`Trainer.init_git_repo` is deprecated and will be removed in v4.34.0 of Transformers. Use `Trainer.init_hf_repo` instead.NT)rrrr)Z clone_fromr .gitignorewzutf-8)encodingz checkpoint-*/ZSM_TRAINING_ENV)!rrrrrrrrrgrrrrrrepoEnvironmentErrorZoverwrite_output_dirr[r\Zgit_pullrrrr: hub_strategyrPALL_CHECKPOINTSopen writelinesenvironr_add_sm_patterns_to_gitignorer)rrrrwriterrrr init_git_repo s>   zTrainer.init_git_repo languagelicensetagsr@finetuned_fromtasks dataset_tagsr. dataset_argsc Csd|s dStj|||||||||| d } | } ttj|jj dd} | | W5QRXdS)a Creates a draft of a model card using the information available to the `Trainer`. Args: language (`str`, *optional*): The language of the model (if applicable) license (`str`, *optional*): The license of the model. Will default to the license of the pretrained model used, if the original model given to the `Trainer` comes from a repo on the Hub. tags (`str` or `List[str]`, *optional*): Some tags to be included in the metadata of the model card. model_name (`str`, *optional*): The name of the model. finetuned_from (`str`, *optional*): The name of the model used to fine-tune this one (if applicable). Will default to the name of the repo of the original model given to the `Trainer` (if it comes from the Hub). tasks (`str` or `List[str]`, *optional*): One or several task identifiers, to be included in the metadata of the model card. dataset_tags (`str` or `List[str]`, *optional*): One or several dataset tags, to be included in the metadata of the model card. dataset (`str` or `List[str]`, *optional*): One or several dataset identifiers, to be included in the metadata of the model card. dataset_args (`str` or `List[str]`, *optional*): One or several dataset arguments, to be included in the metadata of the model card. Nr(z README.mdr) rr&Z from_trainerZ to_model_cardr"rrr:rrwrite) rr)r*r+r@r,r-r.r.r/Ztraining_summaryZ model_cardrlrrrcreate_model_card s"% zTrainer.create_model_cardc Cs|r|jjtjkrdS|jjs:|jdk r:|js:dS|jj}t t t g}t rb| tttg|D]:}tjtj||rfttj||tj||qf|jdk r|j|t|jtj|t|jjtjkrd|jj }ndt!|jj"}t#|j$|||jj%dddgd}|g}|jjtj&tj'fkr|jjtj&krNdnt(|j)}t#|j$|||d|jj%dd } |*| |jdks|jrt+||_n|jj, |dS) NzTraining in progress, step zTraining in progress, epoch Tr**/*r folder_pathrr run_as_futureignore_patternszlast-checkpointz , checkpoint)rr4 path_in_reporrr5)-rrr rPZENDZhub_always_pushris_donerrgrkriruextendrdrfrerrr:r:r[rrrrrrZ save_strategyrQZSTEPSr rrr9rrr CHECKPOINTr!rrgrrljobs) rrrZmodeling_filesZ modeling_filerZmodel_push_jobZ push_jobsr7Zcheckpoint_pushrrrrBsR "     zTrainer._push_from_checkpointcCs:t|dsdS|jdk r6|js6td|jdS)Nrz\Waiting for the current checkpoint push to be finished, this might take a couple of minutes.)rrr8rrZwait_until_donerrrrr]{s   zTrainer._finish_current_pushEnd of training)rblockingr?cKs|dd}|dkrJ|jjrJ|jjdkr8t|jjj}n|jjdd}|jdkr\||j dd| stdS|j fd|i|| t |j|jj||jj| ddgd S) u Upload `self.model` and `self.tokenizer` to the 🤗 model hub on the repo `self.args.hub_model_id`. Parameters: commit_message (`str`, *optional*, defaults to `"End of training"`): Message to commit while pushing. blocking (`bool`, *optional*, defaults to `True`): Whether the function should return only when the `git push` has finished. kwargs (`Dict[str, Any]`, *optional*): Additional keyword arguments passed along to [`~Trainer.create_model_card`]. Returns: The URL of the repository where the model was pushed if `blocking=False`, or a `Future` object tracking the progress of the commit if `blocking=True`. r@N/r#Trrr2r3)rrrrrrrgrrrrr1r]rr)rrr=rr@rrrrs(    zTrainer.push_to_hubc#Cs|j}t|std|dk r"|n|j}|jrJ|jdkrJt|ddd\}}|j|jd|d}t |j j dkr||jkr|jr|j |n|j j |dd}|jr||_||jk r||_|jr|j|_|js|jr|jtj|jd }n|jr|jtj|jd }|j} ||} td |d td | td | d} d} d} d}td|j}t|| | d}|sd}t|drt |j!t"r|j!j}t|| |d}t|| |d}t|| |d}|#|j$dkrd|_%||j&_'t(|D]\}}|j)||||d\}}}t*|jdd}|j+r,|,||nd}|dk rd|-| }| dkrR|ntj.| |fdd} |dk r| dkr||n t/| |dd} |dk r| dkr|n t/| |dd} |dk r|dkr|n t/||dd}|j&0||j1|j2|_2|j3dk r|d|j3dkr|4|5| d|sZ|4|5| d|4|5| d|4|5|dd\} } } }q|j$rt|drt6|d|4|5| d|s|4|5| d|4|5| d|4|5|d|7}|s|7nd}|s|7nd}|s|7nd} |j8dk rf|dk rf|dk rf|j+rR|8t9||| d}!n|8t9||d}!ni}!t:|!}!|dk r|;<|!|d<t=|!>D]0}"|"?|d s|!@|"|!|d |"<qtA|||!| d!S)"rz+dataloader must implement a working __len__NrTrFrrrrrr"rr )make_multiple_ofrJrrr)rrrZ eval_lossesZ eval_predsZeval_label_idsZeval_inputs_idsrr'rrrr_r)Brr\rrrrr#rrrrOrrPrrrrrr%rr rrrrIrrrr1rXr8rrrJr<rr?r'rrrrrrrrcatrDrr rrZ add_arrays_gather_and_numpifyrOfinalizerrMrXrrTr+r,rrrL)#rrr/rrrrr_rrIrrrrrrXZeval_losses_gathererr?Zpreds_gathererZlabels_gathererZinputs_gathererrr`r rrrrrZ eval_losspredsr'Z inputs_idsrrrrrrs                 zTrainer.prediction_loopcCsL|dkr dStrt||}n&tr.t|}n|jjtjkrDt|}t |S)r N) r{rGrxrrrrbrr>rFr rrrrAOs  zTrainer._gather_and_numpifyc Cs|s dSddg}tjtj|jjdr\ttj|jjdd}|}W5QRXnd}|}|D].}||krh| dr||7}qh|d|7}qh||krttj|jjdd }t d || |W5QRX|j dtd |js|jd |jdS) z8Add SageMaker Checkpointing patterns to .gitignore file.Nz*.sagemaker-uploadingz*.sagemaker-uploadedrrr0 rz"Writing .gitignore file. Content: g?z'Add *.sagemaker patterns to .gitignore.)rrrrr:rZ local_dirr"readendswithrr{r0Zgit_addrsleepZ is_repo_cleanZ git_commitZgit_push)rpatternsrlZcurrent_contentcontentpatternrrrr%_s,      z%Trainer._add_sm_patterns_to_gitignorecCs(d|jji}tttdkr(d|d<tf|}t|jj|jj|d|_ t |j j dddk |_ t |j j dddk |_ |j r|j j j}|jjd|j|_td r|jjd |j|_|jr|jjrtd |j r$t |jd ddkr$d dlm}|j j j}||jj|_|jj|_|j|jdS)Nr*rFZsync_with_dataloader)dispatch_batchesrgradient_accumulation_pluginrr}rz0.23.0activation_checkpointingzThe activation_checkpointing in FSDP config and the gradient_checkpointing in training arg can't be set to True simultaneously. Please use FSDP's activation_checkpointing logic when using FSDP.rrr)rrDrrrHrrrLrrOrr rrr}rrrrorNr7rrrrrZdeepspeed_configr)rZgrad_acc_kwargsrMr}rZ ds_pluginrrrrsB      z*Trainer.create_accelerator_and_postprocess) NNNNNNNNNrN)N)N)N)N)N)N)F)TN)NNN)NNNNN)N)N)NNrrNN)T)F)NF)N)NN)FN)NNr)Nr])NNr)N)N)F) NNNNNNNNN)r<T)NNr)tr __module__ __qualname____doc__trainer_pt_utilsrrrrrr r'rModulercr rrrrr0rrMrr5r rrqZ OptimizerrZLambdaLRrrrr"r#rr-r>rArrZSamplerrErrRrYr[r^rrerirb staticmethodrrwrcrrrrFrrrrrrrrrr rWrrSrrJr@rr8rOrKrrVrrrrrCrrrrrrrUrJrXrrrRrrLrrrrGrr'r1rr]rrrAr%rrrrrrsS   |   !"  7{1*  +   i X lV 0) z_ O$4() $ ="    H  B  Y   i ; 899  'r)rQrrrrimportlib.metadatarr(r2rrrr[r5rrcollections.abcrpathlibrtypingrrrrrr r r Z integrationsr rrZhuggingface_hub.utilsrr!rrrZtorch.distributed distributedrRZhuggingface_hubrrr packagingrrZtorch.utils.datarrrrr0rZconfiguration_utilsrZdata.data_collatorrrrZ debug_utilsrrZdependency_versions_checkr rr!r"Zintegrations.deepspeedr#r$r%Z modelcardr&Zmodeling_utilsr'r(r)Zmodels.auto.modeling_autor*r+rr,r-Z pytorch_utilsr.r/Ztokenization_utils_baser0Ztrainer_callbackr1r2r3r4r5r6r7rRr8r9r:r;r<r=r>r?r@rArBrCrDrErFrGrHrIZ trainer_utilsrJrKrLrMrNrOrPrQrRrSrTrUrVrWrXrYrZr[r\r]r^r_r`Z training_argsrarbrcrdrerfrgrhrirjrkrlrmrnrorprqrrrsrtrurvrwrxryrzr{r|r}Zutils.quantization_configr~rrZutils.notebookrrrr;Ztorch_xla.core.xla_modelcoreZ xla_modelrVZtorch_xla.debug.metricsr{rrMrZfairscale.nn.data_parallelrrrrZfairscale.nn.wraprZfairscale.optimrZfairscale.optim.grad_scalerrZ!smdistributed.modelparallel.torchZ modelparallelrZsmdistributed.modelparallelZ SMP_VERSIONrrrrrrZsafetensors.torchrZpeftrZ acceleraterrrHrrrrrrrrrZ get_loggerrrrrrrrrrrrrrs  (         $ Pdp