U d_Z@sddlZddlZddlZddlmZmZmZddlZddlm Z m Z m Z ddl m Z dddd d d d d ddddddddddddgZddZejddZd7ee efdddZd dZd8ee efdd!d"dZdd#d$d Zd9ee efeeefd!d%d Zd:ee efeeefd!d&d Zd;ee efdd!d'd Zdee efdd!d*dZd?ee efed!d+dZd@ee efed!d,dZdAee efed!d-dZ dBee efed!d.dZ!dCee efed!d/dZ"dDee efed!d0dZ#d1dZ$dEee efe%ed3d4dZ&dFee efed!d5dZ'dGee efed!d6dZ(dS)HN)AnyDictUnion)is_initialized_get_device_index _lazy_init)Devicecaching_allocator_alloccaching_allocator_deleteset_per_process_memory_fraction empty_cache memory_statsmemory_stats_as_nested_dictreset_accumulated_memory_statsreset_peak_memory_statsreset_max_memory_allocatedreset_max_memory_cachedmemory_allocatedmax_memory_allocatedmemory_reservedmax_memory_reserved memory_cachedmax_memory_cachedmemory_snapshotmemory_summarylist_gpu_processes mem_get_infocCsttjSN)rtorch_CZ_cuda_cudaHostAllocatorr!r!5/tmp/pip-unpacked-wheel-ua33x9lu/torch/cuda/memory.py_host_allocatorsr#ccs&tjz dVW5tjXdSr)rr Z_cuda_lock_mutexZ_cuda_unlock_mutexr!r!r!r" _free_mutexs  r$devicec Cs|dkrtj}t|}|dkr.tj|}t|tjjjrD|j}t|t sVt dtj |tj ||W5QRSQRXdS)aPerforms a memory allocation using the CUDA memory allocator. Memory is allocated for a given device and a stream, this function is intended to be used for interoperability with other frameworks. Allocated memory is released through :func:`~torch.cuda.caching_allocator_delete`. Args: size (int): number of bytes to be allocated. device (torch.device or int, optional): selected device. If it is ``None`` the default CUDA device is used. stream (torch.cuda.Stream or int, optional): selected stream. If is ``None`` then the default stream for the selected device is used. .. note:: See :ref:`cuda-memory-management` for more details about GPU memory management. NzsInvalid type for stream argument, must be `torch.cuda.Stream` or `int` representing a pointer to a exisiting stream)rcudacurrent_devicerZcurrent_stream isinstanceZstreamsZStreamZ cuda_streamint TypeErrorr&r Z$_cuda_cudaCachingAllocator_raw_alloc)sizer&streamr!r!r"r s   cCstj|dS)aDeletes memory allocated using the CUDA memory allocator. Memory allocated with :func:`~torch.cuda.caching_allocator_alloc`. is freed here. The associated device and stream are tracked inside the allocator. Args: mem_ptr (int): memory address to be freed by the allocator. .. note:: See :ref:`cuda-memory-management` for more details about GPU memory management. N)rr Z%_cuda_cudaCachingAllocator_raw_delete)Zmem_ptrr!r!r"r As)r&returncCsbt|dkrtj}t|}t|ts2td|dksB|dkrPtd |tj ||dS)aSet memory fraction for a process. The fraction is used to limit an caching allocator to allocated memory on a CUDA device. The allowed value equals the total visible memory multiplied fraction. If trying to allocate more than the allowed value in a process, will raise an out of memory error in allocator. Args: fraction(float): Range: 0~1. Allowed memory equals total_memory * fraction. device (torch.device or int, optional): selected device. If it is ``None`` the default CUDA device is used. .. note:: In general, the total available free memory is less than the total capacity. Nz3Invalid type for fraction argument, must be `float`rrz.Invalid fraction value: {}. Allowed range: 0~1) rrr'r(rr)floatr+ ValueErrorformatr Z_cuda_setMemoryFraction)fractionr&r!r!r"r Rs  )r.cCstrtjdS)aReleases all unoccupied cached memory currently held by the caching allocator so that those can be used in other GPU application and visible in `nvidia-smi`. .. note:: :func:`~torch.cuda.empty_cache` doesn't increase the amount of GPU memory available for PyTorch. However, it may help reduce fragmentation of GPU memory in certain cases. See :ref:`cuda-memory-management` for more details about GPU memory management. N)rrr Z_cuda_emptyCacher!r!r!r"r ms cs8gfddt|d}d|tS)aX Returns a dictionary of CUDA memory allocator statistics for a given device. The return value of this function is a dictionary of statistics, each of which is a non-negative integer. Core statistics: - ``"allocated.{all,large_pool,small_pool}.{current,peak,allocated,freed}"``: number of allocation requests received by the memory allocator. - ``"allocated_bytes.{all,large_pool,small_pool}.{current,peak,allocated,freed}"``: amount of allocated memory. - ``"segment.{all,large_pool,small_pool}.{current,peak,allocated,freed}"``: number of reserved segments from ``cudaMalloc()``. - ``"reserved_bytes.{all,large_pool,small_pool}.{current,peak,allocated,freed}"``: amount of reserved memory. - ``"active.{all,large_pool,small_pool}.{current,peak,allocated,freed}"``: number of active memory blocks. - ``"active_bytes.{all,large_pool,small_pool}.{current,peak,allocated,freed}"``: amount of active memory. - ``"inactive_split.{all,large_pool,small_pool}.{current,peak,allocated,freed}"``: number of inactive, non-releasable memory blocks. - ``"inactive_split_bytes.{all,large_pool,small_pool}.{current,peak,allocated,freed}"``: amount of inactive, non-releasable memory. For these core statistics, values are broken down as follows. Pool type: - ``all``: combined statistics across all memory pools. - ``large_pool``: statistics for the large allocation pool (as of October 2019, for size >= 1MB allocations). - ``small_pool``: statistics for the small allocation pool (as of October 2019, for size < 1MB allocations). Metric type: - ``current``: current value of this metric. - ``peak``: maximum value of this metric. - ``allocated``: historical total increase in this metric. - ``freed``: historical total decrease in this metric. In addition to the core statistics, we also provide some simple event counters: - ``"num_alloc_retries"``: number of failed ``cudaMalloc`` calls that result in a cache flush and retry. - ``"num_ooms"``: number of out-of-memory errors thrown. The caching allocator can be configured via ENV to not split blocks larger than a defined size (see Memory Management section of the Cuda Semantics documentation). This helps avoid memory framentation but may have a performance penalty. Additional outputs to assist with tuning and evaluating impact: - ``"max_split_size"``: blocks above this size will not be split. - ``"oversize_allocations.{current,peak,allocated,freed}"``: number of over-size allocation requests received by the memory allocator. - ``"oversize_segments.{current,peak,allocated,freed}"``: number of over-size reserved segments from ``cudaMalloc()``. Args: device (torch.device or int, optional): selected device. Returns statistics for the current device, given by :func:`~torch.cuda.current_device`, if :attr:`device` is ``None`` (default). .. note:: See :ref:`cuda-memory-management` for more details about GPU memory management. csRt|tr@t|dkr|d7}|D]\}}|||q&n||fdS)Nr.)r)dictlenitemsappend)prefixobjkv_recurse_add_to_resultresultr!r"r=s   z,memory_stats.._recurse_add_to_resultr%)rsort collections OrderedDict)r&statsr!r<r"r|s F  cCs"ts iSt|dd}tj|S)zNReturns the result of :func:`~torch.cuda.memory_stats` as a nested dictionary.Toptional)rrrr Z_cuda_memoryStatsr%r!r!r"rs cCst|dd}tj|S)a~Resets the "accumulated" (historical) stats tracked by the CUDA memory allocator. See :func:`~torch.cuda.memory_stats` for details. Accumulated stats correspond to the `"allocated"` and `"freed"` keys in each individual stat dict, as well as `"num_alloc_retries"` and `"num_ooms"`. Args: device (torch.device or int, optional): selected device. Returns statistic for the current device, given by :func:`~torch.cuda.current_device`, if :attr:`device` is ``None`` (default). .. note:: See :ref:`cuda-memory-management` for more details about GPU memory management. TrD)rrr Z!_cuda_resetAccumulatedMemoryStatsr%r!r!r"rs cCst|dd}tj|S)aResets the "peak" stats tracked by the CUDA memory allocator. See :func:`~torch.cuda.memory_stats` for details. Peak stats correspond to the `"peak"` key in each individual stat dict. Args: device (torch.device or int, optional): selected device. Returns statistic for the current device, given by :func:`~torch.cuda.current_device`, if :attr:`device` is ``None`` (default). .. note:: See :ref:`cuda-memory-management` for more details about GPU memory management. TrD)rrr Z_cuda_resetPeakMemoryStatsr%r!r!r"rs cCstdtt|dS)aResets the starting point in tracking maximum GPU memory occupied by tensors for a given device. See :func:`~torch.cuda.max_memory_allocated` for details. Args: device (torch.device or int, optional): selected device. Returns statistic for the current device, given by :func:`~torch.cuda.current_device`, if :attr:`device` is ``None`` (default). .. warning:: This function now calls :func:`~torch.cuda.reset_peak_memory_stats`, which resets /all/ peak memory stats. .. note:: See :ref:`cuda-memory-management` for more details about GPU memory management. zytorch.cuda.reset_max_memory_allocated now calls torch.cuda.reset_peak_memory_stats, which resets /all/ peak memory stats.r%warningswarn FutureWarningrr%r!r!r"rs cCstdtt|dS)aResets the starting point in tracking maximum GPU memory managed by the caching allocator for a given device. See :func:`~torch.cuda.max_memory_cached` for details. Args: device (torch.device or int, optional): selected device. Returns statistic for the current device, given by :func:`~torch.cuda.current_device`, if :attr:`device` is ``None`` (default). .. warning:: This function now calls :func:`~torch.cuda.reset_peak_memory_stats`, which resets /all/ peak memory stats. .. note:: See :ref:`cuda-memory-management` for more details about GPU memory management. zvtorch.cuda.reset_max_memory_cached now calls torch.cuda.reset_peak_memory_stats, which resets /all/ peak memory stats.r%rFr%r!r!r"rs cCst|dddS)a`Returns the current GPU memory occupied by tensors in bytes for a given device. Args: device (torch.device or int, optional): selected device. Returns statistic for the current device, given by :func:`~torch.cuda.current_device`, if :attr:`device` is ``None`` (default). .. note:: This is likely less than the amount shown in `nvidia-smi` since some unused memory can be held by the caching allocator and some context needs to be created on GPU. See :ref:`cuda-memory-management` for more details about GPU memory management. r%zallocated_bytes.all.currentrrgetr%r!r!r"r7scCst|dddS)aReturns the maximum GPU memory occupied by tensors in bytes for a given device. By default, this returns the peak allocated memory since the beginning of this program. :func:`~torch.cuda.reset_peak_memory_stats` can be used to reset the starting point in tracking this metric. For example, these two functions can measure the peak allocated memory usage of each iteration in a training loop. Args: device (torch.device or int, optional): selected device. Returns statistic for the current device, given by :func:`~torch.cuda.current_device`, if :attr:`device` is ``None`` (default). .. note:: See :ref:`cuda-memory-management` for more details about GPU memory management. r%zallocated_bytes.all.peakrrJr%r!r!r"rIscCst|dddS)aReturns the current GPU memory managed by the caching allocator in bytes for a given device. Args: device (torch.device or int, optional): selected device. Returns statistic for the current device, given by :func:`~torch.cuda.current_device`, if :attr:`device` is ``None`` (default). .. note:: See :ref:`cuda-memory-management` for more details about GPU memory management. r%zreserved_bytes.all.currentrrJr%r!r!r"r_s cCst|dddS)aReturns the maximum GPU memory managed by the caching allocator in bytes for a given device. By default, this returns the peak cached memory since the beginning of this program. :func:`~torch.cuda.reset_peak_memory_stats` can be used to reset the starting point in tracking this metric. For example, these two functions can measure the peak cached memory amount of each iteration in a training loop. Args: device (torch.device or int, optional): selected device. Returns statistic for the current device, given by :func:`~torch.cuda.current_device`, if :attr:`device` is ``None`` (default). .. note:: See :ref:`cuda-memory-management` for more details about GPU memory management. r%zreserved_bytes.all.peakrrJr%r!r!r"roscCstdtt|dS)z4Deprecated; see :func:`~torch.cuda.memory_reserved`.zGtorch.cuda.memory_cached has been renamed to torch.cuda.memory_reservedr%)rGrHrIrr%r!r!r"rs cCstdtt|dS)z8Deprecated; see :func:`~torch.cuda.max_memory_reserved`.zOtorch.cuda.max_memory_cached has been renamed to torch.cuda.max_memory_reservedr%)rGrHrIrr%r!r!r"rs cCs tjS)a)Returns a snapshot of the CUDA memory allocator state across all devices. Interpreting the output of this function requires familiarity with the memory allocator internals. .. note:: See :ref:`cuda-memory-management` for more details about GPU memory management. )rr Z_cuda_memorySnapshotr!r!r!r"rs F)r& abbreviatedr.c Cstt|dd}t|d}dd}dd}dd |fd d |fd d |fdd|fdd|fdd|fdd|fdd|fg}g}|d|d|d|d|d|d|D]\}}} |dd|fg} |s| d| dd \} } } }| D]\}}|d!|d!}||d"}||d#}||d$}||d%}| d&krV|} |} |} |}|d'|| || | || | || | ||qqd(d)|fd*d+|fg}|D]~\}}} |d|d!}||d"}||d#}||d$}||d%}|d'|| ||| ||| ||| ||q|dd,|d-}|D]\}}|||d!d.<q>d/d0|jf|d1S)2aReturns a human-readable printout of the current memory allocator statistics for a given device. This can be useful to display periodically during training, or when handling out-of-memory exceptions. Args: device (torch.device or int, optional): selected device. Returns printout for the current device, given by :func:`~torch.cuda.current_device`, if :attr:`device` is ``None`` (default). abbreviated (bool, optional): whether to return an abbreviated summary (default: False). .. note:: See :ref:`cuda-memory-management` for more details about GPU memory management. TrDr%cSsVddddddg}|d}|ddD]$}|d kr4qJ|}|d }|d }q$d ||S) NzB ZKBZMBZGBZTBZPBrri iz{:7d} {}r1)szZpref_szprefixesr8 new_prefixr!r!r" _format_sizes z$memory_summary.._format_sizecSsPdddg}|d}|ddD]$}|dkr.qD|}|d}|d}qd||S) N KMrriq iz {:7d} {} rM)ZcntZpref_cntrOr8rPr!r!r" _format_counts  z%memory_summary.._format_countZallocated_byteszAllocated memoryZ active_bytesz Active memoryZreserved_byteszGPU reserved memoryZinactive_split_byteszNon-releasable memoryZ allocationZ Allocationsactivez Active allocssegmentzGPU reserved segmentsZinactive_splitzNon-releasable allocszK===========================================================================z= {_:16} PyTorch CUDA memory summary, device ID {device:<17d} zK---------------------------------------------------------------------------zX {_:9} CUDA OOMs: {num_ooms:<12d} | {_:6} cudaMalloc retries: {num_alloc_retries:<8d} zK Metric | Cur Usage | Peak Usage | Tot Alloc | Tot Freed all)Z large_poolz from large pool)Z small_poolz from small pool)NNNNr3currentpeak allocatedfreedNz {:<21} | {} | {} | {} | {} Zoversize_allocationszOversize allocationsZoversize_segmentszOversize GPU segmentsr?)_r&-|z| |z| )rrr7r1r6replacejoin)r&rLrCrQrUZmetrics_to_displaylinesZ metric_keyZ metric_name formatterZ submetricsZcurrent_prefvalZ peak_prefvalZallocated_prefvalZ freed_prefvalZ submetric_keyZsubmetric_namer8rYrZr[r\Zfmt_dictr:r;r!r!r"rs                               cCsz ddl}Wntk r"YdSXddlm}z |Wn|k rRYdSXt|dd}||}||}g}|d|t|dkr|d |D],}|j d }|d |j d d |ddqd |S)aReturns a human-readable printout of the running processes and their GPU memory use for a given device. This can be useful to display periodically during training, or when handling out-of-memory exceptions. Args: device (torch.device or int, optional): selected device. Returns printout for the current device, given by :func:`~torch.cuda.current_device`, if :attr:`device` is ``None`` (default). rNz.pynvml module not found, please install pynvml)NVMLError_DriverNotLoadedz-cuda driver can't be loaded, is cuda enabled?TrDzGPU:zno processes are runningizprocess z>10dz uses z>12.3fz MB GPU memory ) pynvmlModuleNotFoundErrorrdZnvmlInitrZnvmlDeviceGetHandleByIndexZ$nvmlDeviceGetComputeRunningProcessesr7r5Z usedGpuMemorypidra)r&rfrdhandleZprocsrbpZmemr!r!r"rs(           cCs*|dkrtj}t|}tj|S)aReturns the global free and total GPU memory occupied for a given device using cudaMemGetInfo. Args: device (torch.device or int, optional): selected device. Returns statistic for the current device, given by :func:`~torch.cuda.current_device`, if :attr:`device` is ``None`` (default). .. note:: See :ref:`cuda-memory-management` for more details about GPU memory management. N)rr'r(rZcudartZcudaMemGetInfor%r!r!r"r@s  )NN)N)N)N)N)N)N)N)N)N)N)N)N)N)NF)N)N))rA contextlibrGtypingrrrrr?rrrZ torch.typesr __all__r#contextmanagerr$r*r r r r strrrrrrrrrrrrrrboolrrrr!r!r!r"s`  ""X" |"