U 9%el  @sl ddlZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlZddlZddlZddlmZddlmZddlmZddlmZmZmZmZmZddlmZddlZddlZddl mZ!dd l"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(dd l)m*Z*dd l+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:m;Z;mZ>m?Z?m@Z@mAZAmBZBmCZCmDZDmEZEmFZFmGZGmHZHmIZImJZJmKZKmLZLmMZMmNZNmOZOmPZPmQZQmRZRmSZSmTZTmUZUmVZVmWZWmXZXmYZYmZZZm[Z[m\Z\m]Z]m^Z^m_Z_m`Z`maZambZbmcZcmdZdmeZemfZfmgZgmhZhmiZie,r4dd ljmkZkmlZleLr~dd lmmnZnmoZompZpmqZqmrZrmsZsmtZtmuZuddlvmwZwddlxmyZynezZnezZydZ{dZ|dZ}dZ~dZdZdfddZdfddZedddZedddZedddZed ddZed!ddZed"ddZed#ddZed$ddZed%ddZd&d'Zd(d)Zd*d+Zd,d-Zd.d/Zd0d1Zd2d3Zd4d5Zd6d7Zd8d9Zd:d;Zdd?Zd@fedAdBdCZdDdEZdFdGZdHdIZdJdKZdLdMZdNdOZdPdQZdRdSZdTdUZdVdWZdXdYZdZd[Zd\d]Zd^d_Zd`daZdbdcZdddeZdfdgZdhdiZdjdkZdldmZdndoZdpdqZdrdsZdtduZdvdwZdxdyZdzd{Zd|d}Zd~dZddZddZddZddZddZddZddZddZddZddZe\rddlZdejkrHejdZzeàešZWn4ek rDZzedeśeɂW5dZ[XYnXn>ejʠˡrZdZn,erne`rndZneredrdZndZdejkrejdZzee̡ZWn:ek rZzede̛deɛeɂW5dZ[XYnXndZeYrddlZe8rddlZeѠҡZndZddZddZddZddZddZddZddZddZddZddZddZddZddZddZddZddZddZddZdd„ZddĄZddƄZddȄZddʄZdd̄Zdd΄ZddЄZdd҄ZddԄZddքZdd؄ZddڄZdfdd܄ZddބZddZGdddZGdddeZGdddeZGdddZejddZejeeejfeddddZGdddejZddZejddZiZddZddZddlZGdddZddZdddddfedddZdddddfedddZ ddZ ddZ dfddZ d d Z d d ZGd ddeZdfeedddZGdddZdddfeeeeedddZddfddZddZGdddejZGdddenZdS( N)Mapping)StringIO)Path)IterableIteratorListOptionalUnion)mock)logging)is_clearml_availableis_fairscale_availableis_optuna_availableis_ray_availableis_sigopt_availableis_wandb_available)is_deepspeed_available)>is_accelerate_availableis_apex_availableis_auto_gptq_availableis_bitsandbytes_availableis_bs4_availableis_cv2_availableis_cython_availableis_decord_availableis_detectron2_availableis_essentia_availableis_faiss_availableis_flash_attn_availableis_flax_availableis_fsdp_availableis_ftfy_availableis_ipex_availableis_jieba_availableis_jinja_availableis_jumanpp_availableis_keras_nlp_availableis_levenshtein_availableis_librosa_availableis_natten_availableis_nltk_availableis_onnx_availableis_optimum_availableis_pandas_availableis_peft_availableis_phonemizer_availableis_pretty_midi_availableis_pyctcdecode_availableis_pytesseract_availableis_pytest_available!is_pytorch_quantization_availableis_rjieba_availableis_safetensors_availableis_scipy_availableis_sentencepiece_availableis_seqio_availableis_soundfile_availbleis_spacy_availableis_sudachi_available#is_tensorflow_probability_availableis_tensorflow_text_availableis_tf2onnx_availableis_tf_availableis_timm_availableis_tokenizers_availableis_torch_availableis_torch_bf16_cpu_availableis_torch_bf16_gpu_availableis_torch_neuroncore_availableis_torch_npu_availableis_torch_tensorrt_fx_availableis_torch_tf32_availableis_torch_tpu_availableis_torch_xpu_availableis_torchaudio_availableis_torchdynamo_availableis_torchvision_availableis_vision_available strtobool)AcceleratorState PartialState)Module _get_checker_get_continue_on_failure _get_runner _is_mocked_patch_unwrap_mock_awareget_optionflags import_path)skip) DoctestItemzjulien-c/bert-xsmall-dummyzjulien-c/dummy-unknownzjulien-c/dummy-diff-tokenizerZ__DUMMY_TRANSFORMERS_USER__zhttps://hub-ci.huggingface.coZ%hf_94wBhPGp6KrrTH3KDchhKpRxZwd6dmHWLLFcCs^ztj|}Wntk r&|}Yn4Xz t|}Wn$tk rXtd|dYnX|S)NIf set, z must be yes or no.)osenvironKeyErrorrQ ValueErrorkeydefaultvalue_valuerhY/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/transformers/testing_utils.pyparse_flag_from_envs  rjcCs^ztj|}Wntk r&|}Yn4Xz t|}Wn$tk rXtd|dYnX|S)Nr^z must be a int.)r_r`raintrbrcrhrhriparse_int_from_envs  rlZRUN_SLOW)reZRUN_PT_TF_CROSS_TESTSTZRUN_PT_FLAX_CROSS_TESTSZRUN_CUSTOM_TOKENIZERSZHUGGINGFACE_CO_STAGINGZTF_GPU_MEMORY_LIMITZRUN_PIPELINE_TESTSZRUN_TOOL_TESTSZRUN_THIRD_PARTY_DEVICE_TESTScCsVtrtrtstd|Sz ddl}Wntk rB|YSX|j|SdS)a' Decorator marking a test as a test that control interactions between PyTorch and TensorFlow. PT+TF tests are skipped by default and we can run only them by setting RUN_PT_TF_CROSS_TESTS environment variable to a truthy value and selecting the is_pt_tf_cross_test pytest mark. ztest is PT+TF testrN) _run_pt_tf_cross_testsrDrAunittestr\pytest ImportErrormarkis_pt_tf_cross_test test_caserorhrhrirrs  rrcCsVtrtrtstd|Sz ddl}Wntk rB|YSX|j|SdS)a& Decorator marking a test as a test that control interactions between PyTorch and Flax PT+FLAX tests are skipped by default and we can run only them by setting RUN_PT_FLAX_CROSS_TESTS environment variable to a truthy value and selecting the is_pt_flax_cross_test pytest mark. ztest is PT+FLAX testrN) _run_pt_flax_cross_testsrDr rnr\rorprqis_pt_flax_cross_testrsrhrhrirvs  rvcCsJtstd|Sz ddl}Wntk r6|YSX|j|SdS)z Decorator marking a test as a staging test. Those tests will run using the staging environment of huggingface.co instead of the real model hub. ztest is staging testrN) _run_stagingrnr\rorprqis_staging_testrsrhrhrirxs  rxcCsJtstd|Sz ddl}Wntk r6|YSX|j|SdS)z Decorator marking a test as a pipeline test. If RUN_PIPELINE_TESTS is set to a falsy value, those tests will be skipped. ztest is pipeline testrN)_run_pipeline_testsrnr\rorprqis_pipeline_testrsrhrhrirzs  rzcCsJtstd|Sz ddl}Wntk r6|YSX|j|SdS)zz Decorator marking a test as a tool test. If RUN_TOOL_TESTS is set to a falsy value, those tests will be skipped. ztest is a tool testrN)_run_tool_testsrnr\rorprq is_tool_testrsrhrhrir|s  r|cCsttd|S)z Decorator marking a test as slow. Slow tests are skipped by default. Set the RUN_SLOW environment variable to a truthy value to run them. z test is slow)rn skipUnless_run_slow_testsrtrhrhrislowsrcCstd|S)z Decorator marking a test as too slow. Slow tests are skipped while they're in the process of being fixed. No test should stay tagged as "tooslow" as these will not be tested by the CI. ztest is too slow)rnr\rrhrhritooslow!srcCsttd|S)z Decorator marking a test for a custom tokenizer. Custom tokenizers require additional dependencies, and are skipped by default. Set the RUN_CUSTOM_TOKENIZERS environment variable to a truthy value to run them. ztest of custom tokenizers)rnr}_run_custom_tokenizersrrhrhricustom_tokenizers,srcCsttd|S)z} Decorator marking a test that requires BeautifulSoup4. These tests are skipped when BeautifulSoup4 isn't installed. ztest requires BeautifulSoup4)rnr}rrrhrhri require_bs46srcCsttd|S)zs Decorator marking a test that requires OpenCV. These tests are skipped when OpenCV isn't installed. ztest requires OpenCV)rnr}rrrhrhri require_cv2=srcCsttd|S)z} Decorator marking a test that requires Levenshtein. These tests are skipped when Levenshtein isn't installed. ztest requires Levenshtein)rnr}r(rrhrhrirequire_levenshteinGsrcCsttd|S)zo Decorator marking a test that requires NLTK. These tests are skipped when NLTK isn't installed. ztest requires NLTK)rnr}r+rrhrhri require_nltkQsrcCsttd|S)zu Decorator marking a test that requires accelerate. These tests are skipped when accelerate isn't installed. ztest requires accelerate)rnr}rrrhrhrirequire_accelerate[srz1.12.0) min_versioncCstt|d||S)zi Decorator marking a test that requires fsdp. These tests are skipped when fsdp isn't installed. ztest requires torch version >= )rnr}r!)rtrrhrhri require_fsdpbsrcCsttd|S)zw Decorator marking a test that requires safetensors. These tests are skipped when safetensors isn't installed. ztest requires safetensors)rnr}r7rrhrhrirequire_safetensorsksrcCsttd|S)zm Decorator marking a test that requires rjieba. These tests are skipped when rjieba isn't installed. ztest requires rjieba)rnr}r6rrhrhrirequire_rjiebarsrcCsttd|S)zk Decorator marking a test that requires jieba. These tests are skipped when jieba isn't installed. ztest requires jieba)rnr}r$rrhrhri require_jiebaysrcCsttd|S)zk Decorator marking a test that requires jinja. These tests are skipped when jinja isn't installed. ztest requires jinja)rnr}r%rrhrhri require_jinjasrcCsttd|S)Nztest requires tf2onnx)rnr}r@rrhrhrirequire_tf2onnxsrcCsttd|S)Nztest requires ONNX)rnr}r,rrhrhri require_onnxsrcCsttd|S)zo Decorator marking a test that requires Timm. These tests are skipped when Timm isn't installed. ztest requires Timm)rnr}rBrrhrhri require_timmsrcCsttd|S)zs Decorator marking a test that requires NATTEN. These tests are skipped when NATTEN isn't installed. ztest requires natten)rnr}r*rrhrhrirequire_nattensrcCsttd|S)zu Decorator marking a test that requires PyTorch. These tests are skipped when PyTorch isn't installed. test requires PyTorch)rnr}rDrrhrhri require_torchsrcCsttd|S)z Decorator marking a test that requires Flash Attention. These tests are skipped when Flash Attention isn't installed. ztest requires Flash Attention)rnr}rrrhrhrirequire_flash_attnsrcCsttd|S)zo Decorator marking a test that requires PEFT. These tests are skipped when PEFT isn't installed. ztest requires PEFT)rnr}r/rrhrhri require_peftsrcCsttd|S)z} Decorator marking a test that requires Torchvision. These tests are skipped when Torchvision isn't installed. ztest requires Torchvision)rnr}rOrrhrhrirequire_torchvisionsrcCsttptd|S)z Decorator marking a test that requires PyTorch or TensorFlow. These tests are skipped when neither PyTorch not TensorFlow is installed. z#test requires PyTorch or TensorFlow)rnr}rDrArrhrhrirequire_torch_or_tfsrcCsttd|S)z Decorator marking a test that requires Intel Extension for PyTorch. These tests are skipped when Intel Extension for PyTorch isn't installed or it does not match current PyTorch version. ztest requires Intel Extension for PyTorch to be installed and match current PyTorch version, see https://github.com/intel/intel-extension-for-pytorch)rnr}r#rrhrhri#require_intel_extension_for_pytorchs rcCsttd|S)z Decorator marking a test that requires TensorFlow probability. These tests are skipped when TensorFlow probability isn't installed. z$test requires TensorFlow probability)rnr}r>rrhrhrirequire_tensorflow_probabilitys rcCsttd|S)zu Decorator marking a test that requires torchaudio. These tests are skipped when torchaudio isn't installed. ztest requires torchaudio)rnr}rMrrhrhrirequire_torchaudiosrcCsttd|S)zu Decorator marking a test that requires TensorFlow. These tests are skipped when TensorFlow isn't installed. ztest requires TensorFlow)rnr}rArrhrhri require_tfsrcCsttd|S)zv Decorator marking a test that requires JAX & Flax. These tests are skipped when one / both are not installed ztest requires JAX & Flax)rnr}r rrhrhri require_flaxsrcCsttd|S){ Decorator marking a test that requires SentencePiece. These tests are skipped when SentencePiece isn't installed. ztest requires SentencePiece)rnr}r9rrhrhrirequire_sentencepiecesrcCsttd|S)rztest requires Seqio)rnr}r:rrhrhri require_seqiosrcCsttd|S)zs Decorator marking a test that requires Scipy. These tests are skipped when SentencePiece isn't installed. ztest requires Scipy)rnr}r8rrhrhri require_scipysrcCsttd|S)u Decorator marking a test that requires 🤗 Tokenizers. These tests are skipped when 🤗 Tokenizers isn't installed. ztest requires tokenizers)rnr}rCrrhrhrirequire_tokenizerssrcCsttd|S)z Decorator marking a test that requires tensorflow_text. These tests are skipped when tensroflow_text isn't installed. ztest requires tensorflow_text)rnr}r?rrhrhrirequire_tensorflow_text#srcCsttd|S)zs Decorator marking a test that requires keras_nlp. These tests are skipped when keras_nlp isn't installed. ztest requires keras_nlp)rnr}r'rrhrhrirequire_keras_nlp+srcCsttd|S)zm Decorator marking a test that requires pandas. These tests are skipped when pandas isn't installed. ztest requires pandas)rnr}r.rrhrhrirequire_pandas2srcCsttd|S)zw Decorator marking a test that requires PyTesseract. These tests are skipped when PyTesseract isn't installed. ztest requires PyTesseract)rnr}r3rrhrhrirequire_pytesseract9srcCsttd|S)z Decorator marking a test that requires PyTorch Quantization Toolkit. These tests are skipped when PyTorch Quantization Toolkit isn't installed. z*test requires PyTorch Quantization Toolkit)rnr}r5rrhrhrirequire_pytorch_quantization@s rcCsttd|S)z Decorator marking a test that requires the vision dependencies. These tests are skipped when torchaudio isn't installed. ztest requires vision)rnr}rPrrhrhrirequire_visionJsrcCsttd|S)zi Decorator marking a test that requires ftfy. These tests are skipped when ftfy isn't installed. ztest requires ftfy)rnr}r"rrhrhri require_ftfyRsrcCsttd|S)zk Decorator marking a test that requires SpaCy. These tests are skipped when SpaCy isn't installed. ztest requires spacy)rnr}r<rrhrhri require_spacyYsrcCsttd|S)zm Decorator marking a test that requires decord. These tests are skipped when decord isn't installed. ztest requires decord)rnr}rrrhrhrirequire_decord`srcCs6tstd|Sddl}t|jdkd|S)a Decorator marking a test that requires a multi-GPU setup (in PyTorch). These tests are skipped on a machine without multiple GPUs. To run *only* the multi_gpu tests, assuming all test names contain multi_gpu: $ pytest -sv ./tests -k "multi_gpu" rrNr ztest requires multiple GPUsrDrnr\torchr}cuda device_countrtrrhrhrirequire_torch_multi_gpugsrcCs6tstd|Sddl}t|jdkd|S)zO Decorator marking a test that requires 0 or 1 GPU setup (in PyTorch). rrNztest requires 0 or 1 GPUrrrhrhrirequire_torch_non_multi_gpuvsrcCs6tstd|Sddl}t|jdkd|S)zT Decorator marking a test that requires 0 or 1 or 2 GPU setup (in PyTorch). rrNztest requires 0 or 1 or 2 GPUsrrrhrhrirequire_torch_up_to_2_gpussrcCsttddd|S)zD Decorator marking a test that requires a TPU (in PyTorch). FZ check_deviceztest requires PyTorch TPU)rnr}rKrrhrhrirequire_torch_tpusrcCsttddd|S)zI Decorator marking a test that requires NeuronCore (in PyTorch). Frz test requires PyTorch NeuronCore)rnr}rGrrhrhrirequire_torch_neuroncoresrcCsttd|S)zB Decorator marking a test that requires NPU (in PyTorch). test requires PyTorch NPU)rnr}rHrrhrhrirequire_torch_npusrcCs.tstd|Sttjdkd|S)a Decorator marking a test that requires a multi-NPU setup (in PyTorch). These tests are skipped on a machine without multiple NPUs. To run *only* the multi_npu tests, assuming all test names contain multi_npu: $ pytest -sv ./tests -k "multi_npu" rr ztest requires multiple NPUs)rHrnr\r}rnpurrrhrhrirequire_torch_multi_npusrcCsttd|S)z Decorator marking a test that requires XPU and IPEX. These tests are skipped when Intel Extension for PyTorch isn't installed or it does not match current PyTorch version. z$test requires IPEX and an XPU device)rnr}rLrrhrhrirequire_torch_xpusrcCs.tstd|Sttjdkd|S)a' Decorator marking a test that requires a multi-XPU setup with IPEX and atleast one XPU device. These tests are skipped on a machine without IPEX or multiple XPUs. To run *only* the multi_xpu tests, assuming all test names contain multi_xpu: $ pytest -sv ./tests -k "multi_xpu" z-test requires IPEX and atleast one XPU devicer ztest requires multiple XPUs)rLrnr\r}rxpurrrhrhrirequire_torch_multi_xpusrZTRANSFORMERS_TEST_DEVICEzUUnknown testing device specified by environment variable `TRANSFORMERS_TEST_DEVICE`: rrrcpuZTRANSFORMERS_TEST_BACKENDz.Failed to import `TRANSFORMERS_TEST_BACKEND` 'zf'! This should be the name of an installed module. The original error (look up to see its traceback): cCsttd|S)z2Decorator marking a test that requires TorchDynamoztest requires TorchDynamo)rnr}rNrrhrhrirequire_torchdynamosrcCsttd|S)z8Decorator marking a test that requires Torch-TensorRT FXztest requires Torch-TensorRT FX)rnr}rIrrhrhrirequire_torch_tensorrt_fxsrcCsttdkd|S)z8Decorator marking a test that requires CUDA and PyTorch.rztest requires CUDA)rnr} torch_devicerrhrhrirequire_torch_gpusrcCsttd|S)zbDecorator marking a test that requires torch>=1.10, using Ampere GPU or newer arch with cuda>=11.0zItest requires torch>=1.10, using Ampere GPU or newer arch with cuda>=11.0)rnr}rFrrhrhrirequire_torch_bf16_gpus rcCsttd|S)z>Decorator marking a test that requires torch>=1.10, using CPU.z$test requires torch>=1.10, using CPU)rnr}rErrhrhrirequire_torch_bf16_cpus rcCsttd|S)z[Decorator marking a test that requires Ampere or a newer GPU arch, cuda>=11 and torch>=1.7.zAtest requires Ampere or a newer GPU arch, cuda>=11 and torch>=1.7)rnr}rJrrhrhrirequire_torch_tf32s rcCsttd|S)z2Decorator marking a test that requires detectron2.ztest requires `detectron2`)rnr}rrrhrhrirequire_detectron2srcCsttd|S)z-Decorator marking a test that requires faiss.ztest requires `faiss`)rnr}rrrhrhri require_faiss"srcCsttd|S)zs Decorator marking a test that requires optuna. These tests are skipped when optuna isn't installed. ztest requires optuna)rnr}rrrhrhrirequire_optuna'srcCsttd|S)zw Decorator marking a test that requires Ray/tune. These tests are skipped when Ray/tune isn't installed. ztest requires Ray/tune)rnr}rrrhrhri require_ray1srcCsttd|S)zs Decorator marking a test that requires SigOpt. These tests are skipped when SigOpt isn't installed. ztest requires SigOpt)rnr}rrrhrhrirequire_sigopt;srcCsttd|S)zq Decorator marking a test that requires wandb. These tests are skipped when wandb isn't installed. ztest requires wandb)rnr}rrrhrhri require_wandbEsrcCsttd|S)zp Decorator marking a test requires clearml. These tests are skipped when clearml isn't installed. ztest requires clearml)rnr}r rrhrhrirequire_clearmlOsrcCsttd|S)zx Decorator marking a test that requires soundfile These tests are skipped when soundfile isn't installed. ztest requires soundfile)rnr}r;rrhrhrirequire_soundfileYsrcCsttd|S)z: Decorator marking a test that requires deepspeed ztest requires deepspeed)rnr}rrrhrhrirequire_deepspeedcsrcCsttd|S)z: Decorator marking a test that requires fairscale ztest requires fairscale)rnr}rrrhrhrirequire_fairscalejsrcCsttd|S)z5 Decorator marking a test that requires apex ztest requires apex)rnr}rrrhrhri require_apexqsrcCsttd|S)z7 Decorator for bits and bytes (bnb) dependency ztest requires bnb)rnr}rrrhrhrirequire_bitsandbytesxsrcCsttd|S)z* Decorator for optimum dependency ztest requires optimum)rnr}r-rrhrhrirequire_optimumsrcCsttd|S)z, Decorator for auto_gptq dependency ztest requires auto-gptq)rnr}rrrhrhrirequire_auto_gptqsrcCsttd|S)z; Decorator marking a test that requires phonemizer ztest requires phonemizer)rnr}r0rrhrhrirequire_phonemizersrcCsttd|S)z< Decorator marking a test that requires pyctcdecode ztest requires pyctcdecode)rnr}r2rrhrhrirequire_pyctcdecodesrcCsttd|S)z8 Decorator marking a test that requires librosa ztest requires librosa)rnr}r)rrhrhrirequire_librosasrcCsttd|S)z9 Decorator marking a test that requires essentia ztest requires essentia)rnr}rrrhrhrirequire_essentiasrcCsttd|S)z< Decorator marking a test that requires pretty_midi ztest requires pretty_midi)rnr}r1rrhrhrirequire_pretty_midisrcCst|dk SN)shutilwhich)cmdrhrhri cmd_existssrcCsttdd|S)z@ Decorator marking a test that requires `/usr/bin/time` /usr/bin/timeztest requires /usr/bin/time)rnr}rrrhrhrirequire_usr_bin_timesrcCsttd|S)z8 Decorator marking a test that requires sudachi ztest requires sudachi)rnr}r=rrhrhrirequire_sudachisrcCsttd|S)8 Decorator marking a test that requires jumanpp ztest requires jumanpp)rnr}r&rrhrhrirequire_jumanppsrcCsttd|S)rztest requires cython)rnr}rrrhrhrirequire_cythonsrcCsTtrddl}|jStr6ddl}t|jdSt rLddl }|SdSdS)z^ Return the number of available gpus (regardless of whether torch, tf or jax is used) rNZGPU) rDrrrrA tensorflowlenconfigZlist_physical_devicesr jax)rtfrrhrhri get_gpu_counts rcCsVtdd}tjtj|}|ds>> from transformers import logging >>> from transformers.testing_utils import CaptureLogger >>> msg = "Testing 1, 2, 3" >>> logging.set_verbosity_info() >>> logger = logging.get_logger("transformers.models.bart.tokenization_bart") >>> with CaptureLogger(logger) as cl: ... logger.info(msg) >>> assert cl.out, msg + " " ``` cCs&||_t|_t|j|_d|_dS)Nr)loggerrior StreamHandlershr)rr(rhrhrir szCaptureLogger.__init__cCs|j|j|Sr)r( addHandlerr+rrhrhrirszCaptureLogger.__enter__cGs|j|j|j|_dSr)r( removeHandlerr+r)rr)rrrhrhrirszCaptureLogger.__exit__cCsd|jdS)Nz captured: r)rrrhrhrirszCaptureLogger.__repr__Nrrhrhrhrir's r'c cs.t}zt|dVW5t|XdS)aT This is a context manager to temporarily change transformers modules logging level to the desired value and have it restored to the original setting at the end of the scope. Example: ```python with LoggingLevel(logging.INFO): AutoModel.from_pretrained("gpt2") # calls logger.info() several times ``` N)transformers_loggingZ get_verbosity set_verbosity)levelZ orig_levelrhrhri LoggingLevels   r1)rreturnc cs6t|}ztjd|dVW5tj|XdS)z Temporary add given path to `sys.path`. Usage : ```python with ExtendSysPath("/path/to/dir"): mymodule = importlib.import_module("mymodule") ``` rN)r_fspathr rremoveinsert)rrhrhri ExtendSysPaths   r6c@seZdZdZddZeddZeddZedd Zed d Z ed d Z eddZ eddZ eddZ eddZeddZeddZeddZddZd%dd Zd!d"Zd#d$ZdS)& TestCasePlusa This class extends *unittest.TestCase* with additional features. Feature 1: A set of fully resolved important file and dir path accessors. In tests often we need to know where things are relative to the current test file, and it's not trivial since the test could be invoked from more than one directory or could reside in sub-directories with different depths. This class solves this problem by sorting out all the basic paths and provides easy accessors to them: - `pathlib` objects (all fully resolved): - `test_file_path` - the current test file path (=`__file__`) - `test_file_dir` - the directory containing the current test file - `tests_dir` - the directory of the `tests` test suite - `examples_dir` - the directory of the `examples` test suite - `repo_root_dir` - the directory of the repository - `src_dir` - the directory of `src` (i.e. where the `transformers` sub-dir resides) - stringified paths---same as above but these return paths as strings, rather than `pathlib` objects: - `test_file_path_str` - `test_file_dir_str` - `tests_dir_str` - `examples_dir_str` - `repo_root_dir_str` - `src_dir_str` Feature 2: Flexible auto-removable temporary dirs which are guaranteed to get removed at the end of test. 1. Create a unique temporary dir: ```python def test_whatever(self): tmp_dir = self.get_auto_remove_tmp_dir() ``` `tmp_dir` will contain the path to the created temporary dir. It will be automatically removed at the end of the test. 2. Create a temporary dir of my choice, ensure it's empty before the test starts and don't empty it after the test. ```python def test_whatever(self): tmp_dir = self.get_auto_remove_tmp_dir("./xxx") ``` This is useful for debug when you want to monitor a specific directory and want to make sure the previous tests didn't leave any data in there. 3. You can override the first two options by directly overriding the `before` and `after` args, leading to the following behavior: `before=True`: the temporary dir will always be cleared at the beginning of the test. `before=False`: if the temporary dir already existed, any existing files will remain there. `after=True`: the temporary dir will always be deleted at the end of the test. `after=False`: the temporary dir will always be left intact at the end of the test. Note 1: In order to run the equivalent of `rm -r` safely, only subdirs of the project repository checkout are allowed if an explicit `tmp_dir` is used, so that by mistake no `/tmp` or similar important part of the filesystem will get nuked. i.e. please always pass paths that start with `./` Note 2: Each test can register multiple temporary dirs and they all will get auto-removed, unless requested otherwise. Feature 3: Get a copy of the `os.environ` object that sets up `PYTHONPATH` specific to the current test suite. This is useful for invoking external programs from the test suite - e.g. distributed training. ```python def test_whatever(self): env = self.get_env() ```cCsg|_t|j|_t|j}|jd|_dD]*}|j|}|d r2|d r2q^q2|rj||_ nt d|j|j d|_ |j d|_ |j d|_dS)Nr)r rrsrcrz+can't figure out the root of the repo from examples)teardown_tmp_dirsrgetfiler#_test_file_pathrresolveparents_test_file_diris_dir_repo_root_dirrb _tests_dir _examples_dir_src_dir)rruptmp_dirrhrhrisetUp2s    zTestCasePlus.setUpcCs|jSr)r<rrhrhritest_file_pathFszTestCasePlus.test_file_pathcCs t|jSr)strr<rrhrhritest_file_path_strJszTestCasePlus.test_file_path_strcCs|jSr)r?rrhrhri test_file_dirNszTestCasePlus.test_file_dircCs t|jSr)rIr?rrhrhritest_file_dir_strRszTestCasePlus.test_file_dir_strcCs|jSr)rBrrhrhrirVszTestCasePlus.tests_dircCs t|jSr)rIrBrrhrhri tests_dir_strZszTestCasePlus.tests_dir_strcCs|jSr)rCrrhrhri examples_dir^szTestCasePlus.examples_dircCs t|jSr)rIrCrrhrhriexamples_dir_strbszTestCasePlus.examples_dir_strcCs|jSr)rArrhrhri repo_root_dirfszTestCasePlus.repo_root_dircCs t|jSr)rIrArrhrhrirepo_root_dir_strjszTestCasePlus.repo_root_dir_strcCs|jSr)rDrrhrhrisrc_dirnszTestCasePlus.src_dircCs t|jSr)rIrDrrhrhri src_dir_strrszTestCasePlus.src_dir_strcCsZtj}|jg}d|jkr*||jn ||j||ddd ||d<|S)a Return a copy of the `os.environ` object that sets up `PYTHONPATH` correctly, depending on the test suite it's invoked from. This is useful for invoking external programs from the test suite - e.g. distributed training. It always inserts `./src` first, then `./tests` or `./examples` depending on the test suite type and finally the preset `PYTHONPATH` if any (all full resolved paths). z /examples PYTHONPATHr:) r_r`copyrSrLappendrOrMgetr)renvpathsrhrhriget_envvs   zTestCasePlus.get_envNcCs|dk rt|dkrd}|dkr d}t|}|dsFtd|d|dkrd|rdtj|dd|jdddn |dkrd}|dkrd}t }|dkr|j ||S) a Args: tmp_dir (`string`, *optional*): if `None`: - a unique temporary path will be created - sets `before=True` if `before` is `None` - sets `after=True` if `after` is `None` else: - `tmp_dir` will be created - sets `before=True` if `before` is `None` - sets `after=False` if `after` is `None` before (`bool`, *optional*): If `True` and the `tmp_dir` already exists, make sure to empty it right away if `False` and the `tmp_dir` already exists, any existing files will remain there. after (`bool`, *optional*): If `True`, delete the `tmp_dir` at the end of the test if `False`, leave the `tmp_dir` and its contents intact at the end of the test. Returns: tmp_dir(`string`): either the same value as passed via *tmp_dir* or the path to the auto-selected tmp dir NTFz./zI`tmp_dir` can only be a relative path, i.e. `./some/path`, but received `` ignore_errorsr>exist_ok) rr= startswithrbexistsrrmtreemkdirtempfilemkdtempr:rW)rrFbeforeafterrrhrhriget_auto_remove_tmp_dirs*    z$TestCasePlus.get_auto_remove_tmp_dirc Cshtdstdtd|d}t}t||dW5QRXt|jdd dd d }|S) a Runs the passed python one liner (just the code) and returns how much max cpu memory was used to run the program. Args: one_liner_str (`string`): a python one liner code that gets passed to `python -c` Returns: max cpu memory bytes used to run the program. This value is likely to vary slightly from run to run. Requirements: this helper needs `/usr/bin/time` to be installed (`apt install time`) Example: ``` one_liner_str = 'from transformers import AutoModel; AutoModel.from_pretrained("t5-large")' max_rss = self.python_one_liner_max_rss(one_liner_str) ``` rz:/usr/bin/time is required, install with `apt install time`z/usr/bin/time -f %M python -c ''rYrrri) rrbshlexsplitrexecute_subprocess_asyncr[rkrreplace)rZ one_liner_strrcsZmax_rssrhrhripython_one_liner_max_rsss z%TestCasePlus.python_one_liner_max_rsscCs^|jD]}tj|ddqg|_trZttttj D]}d|krDtj |=qDdS)NTr]Z ACCELERATE) r:rrcrrRZ _reset_staterSlistr_r`keys)rrkrhrhritearDowns zTestCasePlus.tearDown)NNN)rrrrrGpropertyrHrJrKrLrrMrNrOrPrQrRrSr[rirrrvrhrhrhrir7s<N             D!r7cKstjtj|S)z this is a convenience wrapper, that allows this :: @mockenv(RUN_SLOW=True, USE_TF=False) def test_something(): run_slow = os.getenv("RUN_SLOW", False) use_tf = os.getenv("USE_TF", False) )r patchdictr_r`)kwargsrhrhrimockenvsr{c /stj|p i}|pg}t|t|Bt@}fdd|D}tfdd|D}z&|fdd|DdVW5|fdd|DXdS) aM Temporarily updates the `os.environ` dictionary in-place. Similar to mockenv The `os.environ` dictionary is updated in-place so that the modification is sure to work in all situations. Args: remove: Environment variables to remove. update: Dictionary of environment variables and values to add/update. csi|]}||qSrhrh.0rurkrhri sz#mockenv_context..c3s|]}|kr|VqdSrrhr|rkrhri sz"mockenv_context..csg|]}|qSrhpopr|rkrhri &sz#mockenv_context..csg|]}|dqSrrr|rkrhrir"sN)r_r`setrt frozensetupdate)r4rZstompedZ update_afterZ remove_afterrhrkrimockenv_context s     rcCs*d}|tkr&|j|dddddt|<dS)z This function is to be called from `conftest.py` via `pytest_addoption` wrapper that has to be defined there. It allows loading both `conftest.py` files at once without causing a failure due to adding the same `pytest` option. z--make-reportsstoreFzSgenerate report files. The value of this option is used as a prefix to report names)actionrehelpr N)pytest_opt_registeredZ addoption)parseroptionrhrhripytest_addoption_shared/src sddlm}t|sd}|j}|}|jj}|j}d|tj dddfddd D}g}|j D]"} | D]} t | d rx| | qxqp|r<|jd d dd t|dd|} d} | dt|D]^\} } | j| kr | t|| d| dq2| | jdd| jdd| jdqW5QRXdd}d|j_t|dd} ||| |_|W5QRXt|dd} ||| |_||W5QRXd|j_t|dd} ||| |_|W5QRXt|d d} ||| |_|W5QRXt|d!d"} ||| |_||W5QRXd"|_t|d#d} ||| |_|W5QRXt|d$d} ||| |_|W5QRX||_||_||j_d%S)&a Generate multiple reports at the end of test suite run - each report goes into a dedicated file in the current directory. The report files are prefixed with the test suite name. This function emulates --duration and -rA pytest arguments. This function is to be called from `conftest.py` via `pytest_terminal_summary` wrapper that has to be defined there. Args: - tr: `terminalreporter` passed from `conftest.py` - id: unique id like `tests` or `examples` that will be incorporated into the final reports filenames - this is needed as some jobs have multiple runs of pytest, so we can't have them overwrite each other. NB: this functions taps into a private _pytest API and while unlikely, it could break should pytest do internal changes - also it calls default internal methods of terminalreporter which can be hijacked by various `pytest-` plugins and interfere. r)create_terminal_writerrzreports/Tr_csi|]}|d|dqS)/z.txtrhr|dirrhrir~bsz0pytest_terminal_summary_main..) durationserrors failures_longfailures_short failures_lineZpassesstats summary_shortwarningsdurationcSs|jSr)rxrhrhrizz.pytest_terminal_summary_main..)rdreverserwg?zslowest durations z durations < z secs were omittedz02.2fzs z<8 rc Ssp|d}|sdS|dd|D]H}||}|jd|dddtdd|jd tjtjB}|j |q"dS) Nfailed=zFAILURES SHORT STACK_T)redboldz.*_ _ _ (_ ){10,}_ _ rr) Z getreportsZ write_sepZ_getfailureheadlinerrZ longreprtextrS_twline)trZreportsreprZlongreprrhrhrisummary_failures_shorts   z.summary_failures_shortautorrrrrrZwPpsxXEfrrN)Z_pytest.configrrrZget_terminal_writerrZtbstyleZ reportcharsrrdrvalueshasattrrWsortopenr enumeraterwhenZnodeidrZsummary_failuresZsummary_errorsZsummary_warningsZshort_test_summaryZ summary_stats)ridrrZ orig_writerZ orig_tbstyleZorig_reportcharsZ report_filesZdlistZreplistrfZ durations_minirrhrripytest_terminal_summary_mainBsp      2        rc@seZdZddZdS) _RunOutputcCs||_||_||_dSr) returncoder r )rrr r rhrhrir sz_RunOutput.__init__N)rrrr rhrhrhrirsrcs$|IdH}|r ||qq qdSr)readline)streamcallbackrrhrhri _read_streams rr2cs|rtdd|tj|df|dd|tjjtjj|dIdH}ggd fdd tjt|jfd d t|j fd d g|d IdHt |IdHS)Nz Running: rrr )stdinr r rYrcs.|d}||s*t|||ddS)Nutf-8file)decoderstriprWprint)rZsinkpipelabel)quietrhritees z_stream_subprocess..teecs|tjddS)Nzstdout:r)r r l)rrrhrirrz$_stream_subprocess..cs|tjddS)Nzstderr:r)r r r)rrrhrirrtimeout)r) rrasynciocreate_subprocess_exec subprocessPIPEwaitrr r r)rrYrrrechoprh)rrrrri_stream_subprocesss,  rc Cs~t}|t||||||d}d|}|jdkr^d|j} td|d|jd| |jsz|jsztd|d|S) N)rYrrrrrrrrjz' failed with returncode z, The combined stderr from workers follows: z' produced no output.) rget_event_looprun_until_completerrrr  RuntimeErrorr ) rrYrrrrloopresultZcmd_strr rhrhriros    rocCs*tjdd}tdd|dtj}t|S)z Returns an int value of worker's numerical id under `pytest-xdist`'s concurrent workers `pytest -n N` regime, or 0 if `-n 1` or `pytest-xdist` isn't being used. ZPYTEST_XDIST_WORKERZgw0z^gwrr)r_r`rXrrrrk)Zworkerrhrhripytest_xdist_worker_idsrcCsd}t}||S)z Returns a port number that can be fed to `torch.distributed.launch`'s `--master_port` argument. Under `pytest-xdist` it adds a delta number based on a worker id so that concurrent tests don't try to use the same port at once. i.csg|]}t|qSrhrrrrhrir<scs"i|]\}}t|t|qSrhr)r}ruvrrhrir~@sz#nested_simplify..zNot supported: )numpy isinstancerstupleZndarrayrtolistritemsrIrkZint64rDrZTensorrArZ is_tensorfloatroundZint32Zfloat32r Exceptiontype)objrnprhrrir2s,       rc Cst|d}|}t|dkr2|ddkstnrt|dksBt|ddksVt|ddD],}t|dt|d}|dksbtqb|dd kstW5QRXdS) Nrr rz{}r{rr})r readlinesrrstriplstrip) file_pathrlinesrZ left_indentrhrhri"check_json_file_has_correct_formatQs  rcCst|tjjr|S||fSr)r collectionsabcrrrhrhri to_2tuplebsrc@s eZdZdS)SubprocessCallExceptionN)rrrrhrhrhririsr)commandc Cs~z2tj|tjd}|r0t|dr*|d}|WSWnFtjk rx}z&tdd|d|j|W5d}~XYnXdS)z Runs `command` with `subprocess.check_output` and will potentially return the `stdout`. Will also properly capture if an error occured while running `command` )r rrz Command `rz$` failed with the following error: N) r check_outputSTDOUTrrCalledProcessErrorrroutput)rZ return_stdoutrerhrhri run_commandms   rc@s(eZdZdZddZddZddZdS) RequestCounterz@ Helper class that will count all requests made online. csHd_d_d_tjjj_t _ j j _ fddtjj_S)NrcsjSr)sessionrhrrhrirrz*RequestCounter.__enter__..) head_request_countget_request_countother_request_counthuggingface_hubutils_http get_sessionold_get_sessionrequestsSessionr new_requestrequestrrhrrirs   zRequestCounter.__enter__cOs|jtjj_dSr)rr r r r)rargsrzrhrhrirszRequestCounter.__exit__cKsR|dkr|jd7_n&|dkr0|jd7_n|jd7_tjfd|i|S)NGETr HEADmethod)r rr rr)rrrzrhrhrirs zRequestCounter.new_requestN)rrrrrrrrhrhrhrir~s r) max_attemptswait_before_retry descriptioncsfdd}|S)a To decorate flaky tests. They will be retried on failures. Args: max_attempts (`int`, *optional*, defaults to 5): The maximum number of attempts to retry the flaky test. wait_before_retry (`float`, *optional*): If provided, will wait that number of seconds before retrying the test. description (`str`, *optional*): A string to describe the situation (what / where / why is flaky, link to GH issue/PR comments, errors, etc.) cstfdd}|S)Nc sd}|krzz ||WStk rv}z@td|d|ddtjddk r^t|d7}W5d}~XYqXq||S)Nr zTest failed with z at try r.r)rrr r timesleep)rrzZ retry_countr)r test_func_refrrhriwrappers " z,is_flaky..decorator..wrapper) functoolswraps)rr rr)rri decoratorszis_flaky..decoratorrh)rrrr$rhr#riis_flakysr%c Cs|dkrttjdd}d}t|}|d}|d}|j||d|j ||||fd}| z|j|d} | Wn4t k r} z| || W5d} ~ XYnX|j|d| ddk r|| ddS) a To run a test in a subprocess. In particular, this can avoid (GPU) memory issue. Args: test_case (`unittest.TestCase`): The test that will run `target_func`. target_func (`Callable`): The function implementing the actual testing logic. inputs (`dict`, *optional*, defaults to `None`): The inputs that will be passed to `target_func` through an (input) queue. timeout (`int`, *optional*, defaults to `None`): The timeout (in seconds) that will be passed to the input and output queues. If not specified, the env. variable `PYTEST_TIMEOUT` will be checked. If still `None`, its value will be set to `600`. NZPYTEST_TIMEOUTiXspawnr r)targetrerror)rkr_r`rXmultiprocessingZ get_contextQueueZ JoinableQueueputProcessstart task_doner terminateZfailr) rtZ target_funcinputsrZ start_methohdctxZ input_queueZ output_queueprocessresultsrrhrhrirun_test_in_subprocesss$       r4cCsd}ttj|tjtjBd|}d}t|D]R\}}d|krXd|krXtdd|||<d|kshd |kr.td |r.|r.d }qq.d }|sd |}|S) aPrepare a docstring or a `.md` file to be run by doctest. The argument `string` would be the whole file content if it is a `.md` file. For a python file, it would be one of its docstring. In each case, it may contain multiple python code examples. If `skip_cuda_tests` is `True` and a cuda stuff is detective (with a heuristic), this method will return an empty string so no doctest will be run for `string`. z1(```(?:python|py)\s*\n\s*>>> )((?:.*?\n)*?.*?```))flagsFz load_dataset(z# doctest: +IGNORE_RESULTz(>>> .*load_dataset\(.*)z\1 # doctest: +IGNORE_RESULTz>>>z...zcuda|to\(0\)|device=0Tr) rrncompile MULTILINEDOTALLrrsearchr)stringskip_cuda_testsZcodeblock_patternZ codeblocksZ is_cuda_foundrZ codeblockZmodified_stringrhrhripreprocess_strings(  r<csReZdZUdZedejejBZe e j ddZ e ed<d fdd ZZS) HfDocTestParsera Overwrites the DocTestParser from doctest to properly parse the codeblocks that are formatted with black. This means that there are no extra lines at the end of our snippets. The `# doctest: +IGNORE_RESULT` marker is also added anywhere a `load_dataset` call is made as a print would otherwise fail the corresponding line. Tests involving cuda are skipped base on a naive pattern that should be updated if it is not enough. a # Source consists of a PS1 line followed by zero or more PS2 lines. (?P (?:^(?P [ ]*) >>> .*) # PS1 line (?:\n [ ]* \.\.\. .*)*) # PS2 lines \n? # Want consists of any non-blank lines that do not start with PS1. (?P (?:(?![ ]*$) # Not a blank line (?![ ]*>>>) # Not a line starting with PS1 # !!!!!!!!!!! HF Specific !!!!!!!!!!! (?:(?!```).)* # Match any character except '`' until a '```' is found (this is specific to HF because black removes the last line) # !!!!!!!!!!! HF Specific !!!!!!!!!!! (?:\n|$) # Match a new line or end of string )*) ZSKIP_CUDA_DOCTESTFr;cst||j}t||S)z Overwrites the `parse` method to incorporate a skip for CUDA tests, and remove logs and dataset prints before calling `super().parse` )r<r;r parse)rr:namer"rhrir?8s zHfDocTestParser.parse)r>)rrrrrr6r7VERBOSEZ _EXAMPLE_REboolr_r`rXr;__annotations__r?r%rhrhr"rir=s  r=c@s"eZdZdZeedddZdS)HfDoctestModulez Overwrites the `DoctestModule` of the pytest package to make sure the HFDocTestParser is used when discovering tests. rccsGdddtj}|jjdkrB|jjj|j|jd|jjd}nTz t |j|jj|jdd}Wn2t k r|j drt d|jnYnX|t d }t|}td |tt|jd }|||jD] }|jrtj||j||d VqdS) Ncs2eZdZdZfddZddfdd ZZS)z7HfDoctestModule.collect..MockAwareDocTestFinderzA hackish doctest finder that overrides stdlib internals to fix a stdlib bug. https://github.com/pytest-dev/pytest/issues/3456 https://bugs.python.org/issue25532 cs8t|trt|d|}t|dr*t|}t||S)a3Doctest code does not take into account `@property`, this is a hackish way to fix it. https://bugs.python.org/issue17446 Wrapped Doctests will need to be unwrapped so the correct line number is returned. This will be reported upstream. #8796 fget __wrapped__)rrwgetattrrrunwrapr  _find_lineno)rr source_linesr"rhrirINs    zDHfDoctestModule.collect..MockAwareDocTestFinder._find_linenoNrc s:t|r dStt|||||||W5QRXdSr)rXrYr _find)rrrr@modulerJZglobsseenr"rhrirKbsz=HfDoctestModule.collect..MockAwareDocTestFinder._find)rrrrrIrKr%rhrhr"riMockAwareDocTestFinderHs rNz conftest.pyZ importmode)rootpath)rootmodeZdoctest_ignore_import_errorszunable to import module %r)rF)verbose optionflagscheckerZcontinue_on_failure)r@runnerZdtest)doctestZ DocTestFinderrr@rZ pluginmanagerZ_importconftestZ getoptionrOr[rprr\r=rZrWrUrVrrr9r]Z from_parent)rrNrLfinderrSrUtestrhrhricollectGs8#     zHfDoctestModule.collectN)rrrrrr]rYrhrhrhrirDAsrD(r contextlibrVr! importlibrr r)r_rrmrrr rerrncollections.abcrr)rpathlibrtypingrrrrr r r rZ transformersr.Z integrationsr rrrrrZintegrations.deepspeedrr rrrrrrrrrrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=r>r?r@rArBrCrDrErFrGrHrIrJrKrLrMrNrOrPrQZaccelerate.staterRrSZ_pytest.doctestrTrUrVrWrXrYrZr[Z_pytest.outcomesr\ror]objectZSMALL_MODEL_IDENTIFIERZDUMMY_UNKNOWN_IDENTIFIERZDUMMY_DIFF_TOKENIZER_IDENTIFIERUSERZENDPOINT_STAGINGZTOKENrjrlr~rmrurrwZ_tf_gpu_memory_limitryr{Z_run_third_party_device_testsrrrvrxrzr|rrrrrrrrrIrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr`rZdevicerrrrZ is_availablebackend import_moduleModuleNotFoundErrorrrrZdefault_backendZ jax_devicerrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr&r'contextmanagerr1PathLiker6ZTestCaser7r{rrrrrrrrrorrrrrrrrrrkrr%r4r<Z DocTestParserr=rDrhrhrhris       B(                                              &k+    !    )"  0$5.