U 9%e@sdZddlZddlZddlZddlZddlZddlZddlZddlm Z m Z ddl m Z m Z ddlmZddlmZmZmZddlmZddlmZmZmZmZmZmZd d lmZmZd d lmZ d d l!m"Z"m#Z#m$Z$m%Z%m&Z&d dl'm(Z(e%rddl)m*Z+e$rddl,m-Z.e"r"ddl/Z/e#r6ddl0m1Z2e3dkrRddl4m5Z6n ddl4m6Z6e&7e8Z9da:e dddddddgZ;egdfeGd!d"d"eZ?Gd#d$d$eZ@Gd%d&d&eZAGd'd(d(eZBGd)d*d*eZCee@ZDdeeDe ... generic function which will be executed in its own separate process - `do_multi_processing`: (`bool`) Whether to run function on separate process or not csJtdfdd }t}t||gt|d}||}||S)N)queuec sTz |}Wn8tk rD}zt|t|d}W5d}~XYnX||dS)NN/A) Exceptionloggererrorprintput)r'argsresulter$e/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/transformers/benchmark/benchmark_utils.py wrapper_funcYs  zMseparate_process_wrapper_fn..multi_process_func..wrapper_func)targetr.)r rliststartgetjoin)r.kwargsr4r'pr/r1r2r3multi_process_funcVs z7separate_process_wrapper_fn..multi_process_funcz Function z" is executed in its own process...N)r*info)r$r%r<r2r1r3separate_process_wrapper_fnLs r>cCstSN)_is_memory_tracing_enabledr2r2r2r3is_memory_tracing_enabledpsrAc@s:eZdZUdZeed<eed<eed<eed<eed<dS)Framea `Frame` is a NamedTuple used to gather the current frame state. `Frame` has the following fields: - 'filename' (string): Name of the file currently executed - 'module' (string): Name of the module currently executed - 'line_number' (int): Number of the line currently executed - 'event' (string): Event that triggered the tracing (default will be "line") - 'line_text' (string): Text of the line in the python script filenamemodule line_numberevent line_textN)__name__ __module__ __qualname____doc__str__annotations__intr2r2r2r3rBus  rBc@s*eZdZUdZeed<eed<eed<dS)UsedMemoryStatea `UsedMemoryState` are named tuples with the following fields: - 'frame': a `Frame` namedtuple (see below) storing information on the current tracing frame (current file, location in current file) - 'cpu_memory': CPU RSS memory state *before* executing the line - 'gpu_memory': GPU used memory *before* executing the line (sum for all GPUs or for only `gpus_to_trace` if provided) frameZ cpu_memoryZ gpu_memoryN)rHrIrJrKrBrMrNr2r2r2r3rOs  rOc@s(eZdZUdZeed<edddZdS)Memoryz `Memory` NamedTuple have a single field `bytes` and you can get a human readable str of the number of mega bytes by calling `__repr__` - `byte` (integer): number of bytes, bytesr&cCstt|jSr?)rLbytes_to_mega_bytesrRselfr2r2r3__repr__szMemory.__repr__N)rHrIrJrKrNrMrLrWr2r2r2r3rQs rQc@s2eZdZUdZeed<eed<eed<eed<dS) MemoryStatea `MemoryState` are namedtuples listing frame + CPU/GPU memory with the following fields: - `frame` (`Frame`): the current frame (see above) - `cpu`: CPU memory consumed at during the current frame as a `Memory` named tuple - `gpu`: GPU memory consumed at during the current frame as a `Memory` named tuple - `cpu_gpu`: CPU + GPU memory consumed at during the current frame as a `Memory` named tuple rPcpugpucpu_gpuN)rHrIrJrKrBrMrQr2r2r2r3rXs  rXc@s>eZdZUdZeeed<eeed<eeed<eed<dS) MemorySummaryau `MemorySummary` namedtuple otherwise with the fields: - `sequential`: a list of `MemoryState` namedtuple (see below) computed from the provided `memory_trace` by subtracting the memory after executing each line from the memory before executing said line. - `cumulative`: a list of `MemoryState` namedtuple (see below) with cumulative increase in memory for each line obtained by summing repeated memory increase for a line if it's executed several times. The list is sorted from the frame with the largest memory consumption to the frame with the smallest (can be negative if memory is released) - `total`: total memory increase during the full tracing as a `Memory` named tuple (see below). Line with memory release (negative consumption) are ignored if `ignore_released_memory` is `True` (default). sequential cumulativecurrenttotalN)rHrIrJrKr rXrMrQr2r2r2r3r\s    r\?)functionr&c sttdddts&tdd}nGfdddt}t\}}|t||}|| z$|| d| }| }WnTt k rt t} | j d d D]} t| jtq|dtd YnX|d ||d ks|dkrq |d}q:|SdS)a@ measures peak cpu memory consumption of a given `function` running the function for at least interval seconds and at most 20 * interval seconds. This function is heavily inspired by: `memory_usage` of the package `memory_profiler`: https://github.com/pythonprofilers/memory_profiler/blob/895c4ac7a08020d66ae001e24067da6dcea42451/memory_profiler.py#L239 Args: - `function`: (`callable`): function() -> ... function without any arguments to measure for which to measure the peak memory - `interval`: (`float`, `optional`, defaults to `0.5`) interval in second for which to measure the memory usage - `device_idx`: (`int`, `optional`, defaults to `None`) device id for which to measure gpu usage Returns: - `max_memory`: (`int`) consumed memory peak in Bytes ) process_idr&cSsTt|}z&t|drdnd}t||d}Wntjk rNtdYnX|S)z measures current cpu memory usage of a given `process_id` Args: - `process_id`: (`int`) process_id for which to measure memory Returns - `memory`: (`int`) consumed memory in Bytes memory_infoZget_memory_inforzError with Psutil.)psutilrhasattrgetattrZ AccessDenied ValueError)rcprocessZ meminfo_attrmemoryr2r2r3get_cpu_memorys z/measure_peak_memory_cpu..get_cpu_memoryzsPsutil not installed, we won't log CPU memory usage. Install Psutil (pip install psutil) to use CPU memory tracing.r(cs8eZdZdZeeedfdd ZfddZZ S)z5measure_peak_memory_cpu..MemoryMeasureProcessz `MemoryMeasureProcess` inherits from `Process` and overwrites its `run()` method. Used to measure the memory usage of a process )rcchild_connectionintervalcs2t||_||_||_d|_|j|_dS)Nr)super__init__rcrm connectionnum_measurements mem_usage)rVrcrlrm) __class__rkr2r3ros  z>measure_peak_memory_cpu..MemoryMeasureProcess.__init__csh|jdd}t|j|j|_|jd7_|r8qH|j|j}q|j|j|j|jdS)NrFr)rpsendmaxrrrcrqpollrm)rVstoprkr2r3run s z9measure_peak_memory_cpu..MemoryMeasureProcess.run) rHrIrJrKrNr floatrory __classcell__r2rx)rsr3MemoryMeasureProcesssr|rT) recursivez Process killed. Error in Processgư> N)rNrr*warningrrosgetpidr7recvrtr)rechildrenkillpidr r9 RuntimeError) rbrm device_idxZ max_memoryr|rlZparent_connectionZ mem_processrqparentchildr2rxr3measure_peak_memory_cpus6!     rline)modules_to_tracemodules_not_to_traceevents_to_trace gpus_to_tracer&c strttntddtrz0t |dkrLt t t n|t Wn(ttjfk rtddYqXtptntddgfddtdaS) u Setup line-by-line tracing to record rss mem (RAM) at each line of a module or sub-module. See `./benchmark.py` for usage examples. Current memory consumption is returned using psutil and in particular is the RSS memory "Resident Set Size” (the non-swapped physical memory the process is using). See https://psutil.readthedocs.io/en/latest/#psutil.Process.memory_info Args: - `modules_to_trace`: (None, string, list/tuple of string) if None, all events are recorded if string or list of strings: only events from the listed module/sub-module will be recorded (e.g. 'fairseq' or 'transformers.models.gpt2.modeling_gpt2') - `modules_not_to_trace`: (None, string, list/tuple of string) if None, no module is avoided if string or list of strings: events from the listed module/sub-module will not be recorded (e.g. 'torch') - `events_to_trace`: string or list of string of events to be recorded (see official python doc for `sys.settrace` for the list of events) default to line - `gpus_to_trace`: (optional list, default None) list of GPUs to trace. Default to tracing all GPUs Return: - `memory_trace` is a list of `UsedMemoryState` for each event (default each line of the traced script). - `UsedMemoryState` are named tuples with the following fields: - 'frame': a `Frame` namedtuple (see below) storing information on the current tracing frame (current file, location in current file) - 'cpu_memory': CPU RSS memory state *before* executing the line - 'gpu_memory': GPU used memory *before* executing the line (sum for all GPUs or for only `gpus_to_trace` if provided) `Frame` is a namedtuple used by `UsedMemoryState` to list the current frame state. `Frame` has the following fields: - 'filename' (string): Name of the file currently executed - 'module' (string): Name of the module currently executed - 'line_number' (int): Number of the line currently executed - 'event' (string): Event that triggered the tracing (default will be "line") - 'line_text' (string): Text of the line in the python script zsPsutil not installed, we won't log CPU memory usage. Install psutil (pip install psutil) to use CPU memory tracing.NzUError while initializing communication with GPU. We won't perform GPU memory tracing.Fzvpy3nvml not installed, we won't log GPU memory usage. Install py3nvml (pip install py3nvml) to use GPU memory tracing.cstsSdk r@ttr&|kr&Stttfr@|kr@Sd|jkrNS|jdttsfSdk rttrkrStttfrtfddDrSdk rttrʈkrʈStttfrtfddDrS|j}|jd}| ds| dr&|dd }t || }t ||||}d }dk rb}|j}d } rtrzttrttD]$} t| } t| } | | j7} qtt||| } | S) z Tracing method executed before running each line in a module or sub-module Record memory allocated in a list with debugging information NrHc3s|]}|kVqdSr?r2.0mnamer2r3 sz8start_memory_tracing..traceit..c3s|]}|kVqdSr?r2rrr2r3rs__file__z.pycz.pyor)r@ isinstancerLr6tuple f_globalsallanyf_linenoendswith linecachegetlinerstriprBrdZrssrtorch_empty_cacher tf_contextrZ _clear_cachesnvmlnvmlInitnvmlDeviceGetHandleByIndexnvmlDeviceGetMemoryInfoused nvmlShutdownrOappend)rPrFr.linenorCrZ traced_statecpu_memZmemgpu_memihandleZmeminfoZ mem_stateZdevicesrZlog_gpu memory_tracerrritraceitrr3rs^   $$        z%start_memory_tracing..traceitT)rrerrrr*rrrrr6rangeZnvmlDeviceGetCountrOSErrorZ NVMLErrorrrsyssettracer@)rrrrr2rr3start_memory_tracingCs0(   L rT)rignore_released_memoryr&c Cszda|dk rvt|dkrvg}g}tdd}t|dd|ddD]\\}}}\}} } | |} | |} | | } |t|t| t| t| d|t|t| t| t| | d||d| 7<||d| 7<||d | 7<qJt|d dd d }d d|D}t|ddd d }|rLt dd|D}nt dd|D}t|}t ||||dSdS)aW Stop memory tracing cleanly and return a summary of the memory trace if a trace is given. Args: `memory_trace` (optional output of start_memory_tracing, default: None): memory trace to convert in summary `ignore_released_memory` (boolean, default: None): if True we only sum memory increase to compute total memory Return: - None if `memory_trace` is None - `MemorySummary` namedtuple otherwise with the fields: - `sequential`: a list of `MemoryState` namedtuple (see below) computed from the provided `memory_trace` by subtracting the memory after executing each line from the memory before executing said line. - `cumulative`: a list of `MemoryState` namedtuple (see below) with cumulative increase in memory for each line obtained by summing repeated memory increase for a line if it's executed several times. The list is sorted from the frame with the largest memory consumption to the frame with the smallest (can be negative if memory is released) - `total`: total memory increase during the full tracing as a `Memory` named tuple (see below). Line with memory release (negative consumption) are ignored if `ignore_released_memory` is `True` (default). `Memory` named tuple have fields - `byte` (integer): number of bytes, - `string` (string): same as human readable string (ex: "3.5MB") `Frame` are namedtuple used to list the current frame state and have the following fields: - 'filename' (string): Name of the file currently executed - 'module' (string): Name of the module currently executed - 'line_number' (int): Number of the line currently executed - 'event' (string): Event that triggered the tracing (default will be "line") - 'line_text' (string): Text of the line in the python script `MemoryState` are namedtuples listing frame + CPU/GPU memory with the following fields: - `frame` (`Frame`): the current frame (see above) - `cpu`: CPU memory consumed at during the current frame as a `Memory` named tuple - `gpu`: GPU memory consumed at during the current frame as a `Memory` named tuple - `cpu_gpu`: CPU + GPU memory consumed at during the current frame as a `Memory` named tuple FNrcSs dddgS)Nrr2r2r2r2r3z%stop_memory_tracing..rrPrYrZr[rrcSs |ddS)Nrrr2xr2r2r3r0rT)keyreversecSs2g|]*\}\}}}t|t|t|t|dqS)r)rXrQ)rrP cpu_mem_inc gpu_mem_inccpu_gpu_mem_incr2r2r3 2s z'stop_memory_tracing..cSs|jjSr?r[rRrr2r2r3r<rcss|]}td|jjVqdS)rN)rur[rRrZ step_tracer2r2r3r?sz&stop_memory_tracing..css|]}|jjVqdSr?rrr2r2r3rAs)r]r^r_r`) r@lenrziprrXrQsorteditemssumr\)rrZmemory_diff_traceZmemory_curr_traceZcumulative_memory_dictrPrrZ next_frameZ next_cpu_memZ next_gpu_memrrrZcumulative_memoryZ total_memoryr2r2r3stop_memory_tracingsf/     r) memory_amountr&cCs|d?S)zLUtility to convert a number of bytes (int) into a number of mega bytes (int)r~r2)rr2r2r3rTOsrTc@sBeZdZUdZeed<eed<eed<d*eedddZe d d Z e e d d Z e ee e ed ddZe ee e ed ddZe ee e eeegd ddZe ee e eeegd ddZedddZedddZeeegdddZeeegdddZdd Ze d!d"Zd#d$Zed%d&d'Zd(d)ZdS)+ Benchmarkz Benchmarks is a simple but feature-complete benchmarking script to compare memory and time performance of models in Transformers. r.configs frameworkN)r.rcCs||_|dkr$dd|jjD|_ntt|jj||_td|jdt|jj rnt ddkrnt dd|_d|_d|_dS)NcSsi|]}|t|qSr2)rZfrom_pretrainedr model_namer2r2r3 asz&Benchmark.__init__..z The class z is deprecated. Hugging Face Benchmarking utils are deprecated in general and it is advised to use external Benchmarking libraries to benchmark Transformer models.Z TRANSFORMERS_USE_MULTIPROCESSINGrzMemory consumption will not be measured accurately if `args.multi_process` is set to `False.` The flag 'TRANSFORMERS_USE_MULTIPROCESSING' should only be disabled for debugging / testing.)r. model_namesZ config_dictdictrwarningswarnrs FutureWarningrjrgetenvr*r _print_fnZ_framework_version_environment_info)rVr.rr2r2r3ro^s"  zBenchmark.__init__cs2jdkr,jjr&fdd}|_nt_jS)Nc s:tjjd}|d|dW5QRXt|dS)Na )openr.Z log_filenamewriter9r,)r.Zlog_filerUr2r3 print_and_log}sz)Benchmark.print_fn..print_and_log)rr.Z log_printr,)rVrr2rUr3print_fnxs   zBenchmark.print_fncCsdSr?r2rUr2r2r3framework_versionszBenchmark.framework_version)r batch_sizesequence_lengthr&cCsdSr?r2rVrrrr2r2r3_inference_speedszBenchmark._inference_speedcCsdSr?r2rr2r2r3 _train_speedszBenchmark._train_speedcCsdSr?r2rr2r2r3_inference_memoryszBenchmark._inference_memorycCsdSr?r2rr2r2r3 _train_memoryszBenchmark._train_memoryrScOst|j|jj||Sr?)r>rr.r%rVr.r:r2r2r3inference_speedszBenchmark.inference_speedcOst|j|jj||Sr?)r>rr.r%rr2r2r3 train_speedszBenchmark.train_speedcOst|j|jj||Sr?)r>rr.r%rr2r2r3inference_memoryszBenchmark.inference_memorycOst|j|jj||Sr?)r>rr.r%rr2r2r3 train_memoryszBenchmark.train_memoryc Csdd|jjD}t|}t|}t|}t|}t|jjD]n\}}||ddt|jj|jj|jjdd|jjDd}t|||<t|||<t|||<t|||<d} } |jjD]} |jjD]} |jj rP|jj r$| || | \} } | ||d| | <|jj rP| || | }|||d| | <|jjr|jj r||| | \} } | ||d| | <|jj r||| | }|||d| | <qqqF|jj r|jj r|dd d d |j|d d |||jj|jjr|d|jj rV|ddd d |j|dd |||jj|jjr|ddd d || |jjrJ|jj r|ddd d ||d |||jj|jjr|d|jj r|ddd d |j|dd |||jj|jjrJ|ddd d || |jjr|ddd d |ddd|jDd|jjrt|jj ddd4}t!"|}|jD]\}}|#||gqW5QRXt$||||| | S)NcSsi|] }|iqSr2r2rr2r2r3rsz!Benchmark.run..rz / cSsi|] }|iqSr2r2)rrr2r2r3rs)bsssr/r/z ====================zINFERENCE - SPEED - RESULT(z====================z Time in s) type_labelzTPU was used for inference. Note that the time after compilation stabilized (after ~10 inferences model.forward(..) calls) was measured.zINFERENCE - MEMORY - RESULTz Memory in MBz,INFERENCE - MEMOMRY - LINE BY LINE - SUMMARYzTRAIN - SPEED - RESULTSzTPU was used for training. Note that the time after compilation stabilized (after ~10 train loss=model.forward(...) + loss.backward() calls) was measured.zTRAIN - MEMORY - RESULTSz(TRAIN - MEMOMRY - LINE BY LINE - SUMMARYzENVIRONMENT INFORMATIONrcSs g|]\}}d|d|qS)z- : r2)rpropvalr2r2r3rsz!Benchmark.run..wr)modenewline)%r.rcopydeepcopy enumeraterrZ batch_sizesZsequence_lengthsZ inferencerjrspeedrZtrainingrrcenter print_results save_to_csvZinference_time_csv_fileis_tpuZinference_memory_csv_fileZtrace_memory_line_by_lineprint_memory_trace_statisticsZtrain_time_csv_fileZtrain_memory_csv_fileZ env_printr9environment_inforrZenv_info_csv_filecsvwriterwriterowr!)rV result_dictZinference_result_timeZinference_result_memoryZtrain_result_timeZtrain_result_memorycrZ model_dictr"r#rrrjtimecsv_filerrvaluer2r2r3rys                         $  z Benchmark.runcCs|jdkri}t|d<|j|d<|jdkr8|jj|d<|jdkrZ|jj|d<|jj|d<|j|d<t |d <t |d <t |d <t d |d <t t |d<t t |d<|jj|d<|jj|d<|jj|d<trttj|d<ntdd|d<|jj|d<|jjrd|d<trtt|jj }t!||d<tt"|j|d<t#|d|d<t$||d<t%n*tdd|d<d|d<d|d<d|d<|jj&|d<||_|jS) NZtransformers_versionrZPyTorchZuse_torchscriptZ TensorFlow eager_modeuse_xlarpython_versionsystemrYr architecturedater fp16Zuse_multiprocessingonly_pretrain_modelZ cpu_ram_mbzyPsutil not installed, we won't log available CPU memory. Install psutil (pip install psutil) to log available CPU memory.r(Zuse_gpurZnum_gpusrZZ gpu_ram_mbZgpu_power_wattsZgpu_performance_statezypy3nvml not installed, we won't log GPU memory usage. Install py3nvml (pip install py3nvml) to log information about GPU.Zuse_tpu)'rversionrr.Z torchscriptr rrplatformrr processorrrrnowr rr%rrrTreZvirtual_memoryr`r*rZis_gpurrrrrZnvmlDeviceGetNamerZ!nvmlDeviceGetPowerManagementLimitZnvmlDeviceGetPerformanceStaterr)rVr=rr2r2r3rs\                  zBenchmark.environment_infoc Cs|d|dddddd|d|d|jjD]}||dD]}||dD]}||d ||}t|trtd |d }|d krd nt|}nt|}||dddt|dt|d|dqjqZqJ|ddS) NzP--------------------------------------------------------------------------------z Model Namez Batch Sizez Seq Lengthrrr/rgz< 0.001)rrr.rrrzroundrL)rVrrrrrr/r2r2r3rJs& &     zBenchmark.print_results)summaryc Cs|dddd|jD|dddd|jddD|ddd d|jd dD|d |jdS) Nz" Line by line memory consumption: rcss6|].}|jjd|jjd|jd|jjVqdS):: mem rNrPrCrEr[rGrstater2r2r3rcsz:Benchmark.print_memory_trace_statistics..z$ Lines with top memory consumption: c ss8|]0}d|jjd|jjd|jd|jjVqdSz=> rrrNr r!r2r2r3rjsz' Lines with lowest memory consumption: c ss8|]0}d|jjd|jjd|jd|jjVqdSr#r r!r2r2r3rqsiz Total memory increase: )rr9r]r^r`)rVrr2r2r3r`s,     z'Benchmark.print_memory_trace_statisticsc Cs|jjs dS|dt|dd}t|jjdkrDtd|jdddg}tj||d gd }| |jjD]\}||d }|D]F}||D]8} ||| } | ||| t | t sd nd | d qqqrW5QRXdS)NzSaving results to csv.r)rrz,At least 1 model should be defined, but got modelrrr/) fieldnamesz{}z{:.4f})r%rrr/)r.rrrrrrhr DictWriter writeheaderrrrzformat) rVrrCr r&rrZresult_dict_modelrrZ result_modelr2r2r3rxs.      zBenchmark.save_to_csv)NN)rHrIrJrKrrMrrLropropertyrrrrNrzrrrQrr\rrrrrrryrrrrr2r2r2r3rTsH     c :r)raN)NNrN)NT)LrKrrrrrrrabcrr collectionsrrrmultiprocessingrrr Zmultiprocessing.connectionr typingr r r rrrrrrrrutilsrrrrrZbenchmark_args_utilsrZ torch.cudarrZtensorflow.python.eagerrrreZpy3nvml.py3nvmlZpy3nvmlrrsignalrr Z get_loggerrHr*r@r!boolr>rArBrOrQrXr\Z MemoryTracerNrrLrrrTrr2r2r2r3s           "$x   t