U -e@sddlZddlmZmZddlmZddlmZmZe e Z GdddZ edd Z d d Zd d ZeejedddZdS)N)PoolRLock)tqdm) experimentalloggingc@seZdZdZdS)ParallelBackendConfigN)__name__ __module__ __qualname__ backend_namer r [/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/datasets/parallel/parallel.pyr srcCs2tjdkrt|||||||St|||||||S)a **Experimental.** Apply a function to iterable elements in parallel, where the implementation uses either multiprocessing.Pool or joblib for parallelization. Args: function (`Callable[[Any], Any]`): Function to be applied to `iterable`. iterable (`list`, `tuple` or `np.ndarray`): Iterable elements to apply function to. num_proc (`int`): Number of processes (if no backend specified) or jobs (using joblib). types (`tuple`): Additional types (besides `dict` values) to apply `function` recursively to their elements. disable_tqdm (`bool`): Whether to disable the tqdm progressbar. desc (`str`): Prefix for the tqdm progressbar. single_map_nested_func (`Callable`): Map function that applies `function` to an element from `iterable`. Takes a tuple of function, data_struct, types, rank, disable_tqdm, desc as input, where data_struct is an element of `iterable`, and `rank` is used for progress bar. N)rr _map_with_multiprocessing_pool_map_with_joblibfunctioniterablenum_proctypes disable_tqdmdescsingle_map_nested_funcr r r parallel_maps rc Csr|t|kr|nt|}g}t|D]d}t||} t||} | |t|| } | | || krddnd} |||| | ||||fq$t|tdd|Dkrtdt|dtdd|Dtd|d t|d d d |Dd \} }|stft j } }t || |d}| ||}W5QRXtd|ddd |D}tdt|d|S)Nrcss|]}t|dVqdSrNlen.0ir r r 3sz1_map_with_multiprocessing_pool..zHError dividing inputs iterable among processes. Total number of objects z , length: css|]}t|dVqdSrrrr r rr!5sz Spawning z processes for z objects in slices of cSsg|]}t|dqS)rrrr r r ;sz2_map_with_multiprocessing_pool..)NN)initargs initializerz Finished z processescSsg|]}|D]}|q qSr r )rZproc_resobjr r rr"Csz Unpacked z objects) rrangeminappendsum ValueErrorloggerinforrZset_lockrmap)rrrrrrrZ split_kwdsindexdivmodstartendr#r$poolZmappedr r rr)s0     "rc sPddljtj|d.fdd|DW5QRSQRXdS)Nr)Zn_jobsc3s(|] }|dddfVqdS)NT)Zdelayed)rr%rjoblibrrr rr!Osz#_map_with_joblib..)r5parallel_backendrr ZParallelrr r4rrIs r)r ccs8|t_|dkr ddlm}|z dVW5dt_XdS)a **Experimental.** Configures the parallel backend for parallelized dataset loading, which uses the parallelization implemented by joblib. Args: backend_name (str): Name of backend for parallelization implementation, has to be supported by joblib. Example usage: ```py with parallel_backend('spark'): dataset = load_dataset(..., num_proc=2) ``` Zsparkr)register_sparkN)rr Z joblibsparkr7)r r7r r rr6Ts  r6) contextlibmultiprocessingrrZ tqdm.autorutilsrrZ get_loggerr r+rrrrcontextmanagerstrr6r r r rs