U -efm@s:dZddlZddlZddlZddlmZmZmZmZm Z m Z ddl Z ddl ZddlmZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZmZdd l m!Z!ddl"m#Z#ddl$m%Z%m&Z&m'Z'ddl(m)Z)ddl*m+Z+m,Z,e)e-Z.Gddde%Z/ddZ0GdddZ1Gddde1Z2dS)z Metrics base class.N)AnyDictListOptionalTupleUnion)config)Dataset) ArrowReader) ArrowWriter)DownloadConfig)DownloadManager)Features) DatasetInfo MetricInfo)camelcase_to_snakecase) deprecated) BaseFileLockFileLockTimeout) get_logger)copyfunc temp_seedcs0eZdZdZfddZddZddZZS) FileFreeLockz-Thread lock until a file **cannot** be lockedcs"t||_tj|f||dSN)rfilelocksuper__init__)self lock_fileargskwargs __class__P/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/datasets/metric.pyr.s zFileFreeLock.__init__cCsHz|jjdddWntk r2|jj|_YnX|jd|_dS)Ng{Gz?g{Gz?)timeoutZpoll_intervall)racquirerr _lock_file_fdreleaserr%r%r&_acquire2s  zFileFreeLock._acquirecCs d|_dSr)r)r+r%r%r&_release=szFileFreeLock._release)__name__ __module__ __qualname____doc__rr,r- __classcell__r%r%r#r&r+s  rcCsPt|tkrt|dkr|Sdd}d||ddd||dddS) NcSsddd|DS)Nz, css|]}t|VqdSr)repr).0xr%r%r& Hsz?summarize_if_long_list..format_chunk..)join)chunkr%r%r& format_chunkGsz,summarize_if_long_list..format_chunk[z, ..., ])typelistlen)objr:r%r%r&summarize_if_long_listCsrCc@seZdZdZedddZeddZeeddd Z ee edd d Z eedd d Z eedddZ eedddZeedddZee edddZeedddZee eedddZee eedddZeedddZee edddZd S)!MetricInfoMixinuThis base class exposes some attributes of MetricInfo at the base level of the Metric for easy access. Use the new library 🤗 Evaluate instead: https://huggingface.co/docs/evaluate )infocCs ||_dSr _metric_info)rrEr%r%r&rYszMetricInfoMixin.__init__cCs|jS)zN:class:`datasets.MetricInfo` object containing all the metadata in the metric.rFr+r%r%r&rE\szMetricInfoMixin.inforeturncCs|jjSr)rG metric_namer+r%r%r&nameaszMetricInfoMixin.namecCs|jjSr)rG experiment_idr+r%r%r&rLeszMetricInfoMixin.experiment_idcCs|jjSr)rG descriptionr+r%r%r&rMiszMetricInfoMixin.descriptioncCs|jjSr)rGcitationr+r%r%r&rNmszMetricInfoMixin.citationcCs|jjSr)rGfeaturesr+r%r%r&rOqszMetricInfoMixin.featurescCs|jjSr)rGinputs_descriptionr+r%r%r&rPusz"MetricInfoMixin.inputs_descriptioncCs|jjSr)rGhomepager+r%r%r&rQyszMetricInfoMixin.homepagecCs|jjSr)rGlicenser+r%r%r&rR}szMetricInfoMixin.licensecCs|jjSr)rG codebase_urlsr+r%r%r&rSszMetricInfoMixin.codebase_urlscCs|jjSr)rGreference_urlsr+r%r%r&rTszMetricInfoMixin.reference_urlscCs|jjSr)rG streamabler+r%r%r&rUszMetricInfoMixin.streamablecCs|jjSr)rGformatr+r%r%r&rVszMetricInfoMixin.formatN)r.r/r0r1rrpropertyrEstrrKrrLrMrNrrOrPrQrRrrSrTboolrUrVr%r%r%r&rDMs8  rDc @sPeZdZdZedd2eeeeeeeeeeeee ee fd d d Z d d Z ddZ ddZd3eeefdddZeeeeefdddZddZddZddZdddeedddZdddd d!Zddd"d#d$Zd4d%d&Zedd'd(Zd5eeeed)d*d+Z d,d-Z!ddde"ee#fdd.d/Z$d0d1Z%dS)6MetricuA Metric is the base class and common API for all metrics. Use the new library 🤗 Evaluate instead: https://huggingface.co/docs/evaluate Args: config_name (``str``): This is used to define a hash specific to a metrics computation script and prevents the metric's data to be overridden when the metric loading script is modified. keep_in_memory (:obj:`bool`): keep all predictions and references in memory. Not possible in distributed settings. cache_dir (``str``): Path to a directory in which temporary prediction/references data will be stored. The data directory should be located on a shared file-system in distributed setups. num_process (``int``): specify the total number of nodes in a distributed settings. This is useful to compute metrics in distributed setups (in particular non-additive metrics like F1). process_id (``int``): specify the id of the current process in a distributed setup (between 0 and num_process-1) This is useful to compute metrics in distributed setups (in particular non-additive metrics like F1). seed (:obj:`int`, optional): If specified, this will temporarily set numpy's random seed when :func:`datasets.Metric.compute` is run. experiment_id (``str``): A specific experiment id. This is used if several distributed evaluations share the same file system. This is useful to compute metrics in distributed setups (in particular non-additive metrics like F1). max_concurrent_cache_files (``int``): Max number of concurrent metrics cache files (default 10000). timeout (``Union[int, float]``): Timeout in second for distributed setting synchronization. uOUse the new library 🤗 Evaluate instead: https://huggingface.co/docs/evaluateNFrr'd) config_namekeep_in_memory cache_dir num_process process_idseedrLmax_concurrent_cache_filesr'c Ks|pd|_|} t|jj| _|j| _|p.d| _t|| t |t rP|dkrXt dt |t rj||krrt d|r|dkrt d||_ ||_ ||_||_tj|ptj|_||_|dkrtj^} }} } | dkr|| n|d|_n||_| |_tt|j ||_ tt|j!||_!tt|j"||_"|j j#j$|j%j&7_$|j!j#j$|j%j&7_$|j"j#j$|j%j&7_$d|_'d|_(d|_)d|_*d|_+d|_,d|_-d|_.d|_/dS) NdefaultZdefault_experimentrz.'process_id' should be a number greater than 0z8'num_process' should be a number greater than process_idrzPUsing 'keep_in_memory' is not possible in distributed setting (num_process > 1).ip)0r]_inforr$r.rJrLrDr isinstanceint ValueErrorr`rarcr^ospath expanduserr ZHF_METRICS_CACHE_data_dir_root_build_data_dirdata_dirnprandomZ get_staterbr'types MethodTypercompute add_batchadd__func__r1rErP buf_writerwriterwriter_batch_sizedatacache_file_namerrendez_vous_lock file_paths filelocks)rr]r^r_r`rarbrLrcr'r"rE_posr%r%r&rsL     zMetric.__init__cCs|jdkrdSt|jS)zReturn the number of examples (predictions or predictions/references pair) currently stored in the metric's cache. Nr)rxrAr+r%r%r&__len__szMetric.__len__c Cs(d|jd|jd|jdt|d S)NzMetric(name: "z ", features: z , usage: """z""", stored examples: ))rKrOrPrAr+r%r%r&__repr__s&zMetric.__repr__cCs,|j}tj||j|j}tj|dd|S)aPath of this metric in cache_dir: Will be: self._data_dir_root/self.name/self.config_name/self.hash (if not none)/ If any of these element is missing or if ``with_version=False`` the corresponding subfolders are dropped. T)exist_ok)rlrirjr8rKr]makedirs)rZbuilder_data_dirr%r%r&rmszMetric._build_data_dirrHcCstj|j|jd|jd|jd}d}t|jD]}t |d}z|j |dWnt k r|jdkrt d|d|jd d||jdkrt d |jd dt t}tj|j|jd|d|jd|jd}Yq6Xqq6||fS) zTCreate a new cache file. If the default cache file is used, we generated a new hash.-.arrowN.lockr'rz^Error in _create_cache_file: another metric instance is already using the local cache file at .. Please specify an experiment_id (currently: :) to avoid collision between distributed metric instances.zCannot acquire lock, too many metric instance are operating concurrently on this file system.You should set a larger value of max_concurrent_cache_files when creating the metric (current value is z).)rirjr8rnrLr`rarangercrr(rrhrXuuiduuid4)rr' file_pathriZ file_uuidr%r%r&_create_cache_file s4(     zMetric._create_cache_filec sjdkr&jdkrtdjg}nfddtjD}g}t|D]t\}}|dkrh|jqJt|d}z|jj dWn,t k rtd |d |d dYqJX||qJ||fS) zGet a lock on all the cache files in a distributed setup. We wait for timeout second to let all the distributed node finish their tasks (default is 100 seconds). rNz|Metric cache file doesn't exist. Please make sure that you call `add` or `add_batch` at least once before calling `compute`.c s2g|]*}tjjjdjd|dqS)rrrirjr8rnrLr`r5rar+r%r& 6sz/Metric._get_all_cache_files..rrrz#Cannot acquire lock on cached file z for process .) r`r{rhr enumerateappendrrr(r'r)rr}r~rarrr%r+r&_get_all_cache_files*s0      zMetric._get_all_cache_filesc sxfddtjD}|D]V}t|}z|jjdWn.tk rhtd|djddYqX|qdS)Nc s2g|]*}tjjjdjd|dqS)rz .arrow.lockrrr+r%r&rNsz5Metric._check_all_processes_locks..rExpected to find locked file  from process  but it doesn't exist.) rr`rr(r'rrhrar*)rZexpected_lock_file_namesexpected_lock_file_name nofilelockr%r+r&_check_all_processes_locksMs z!Metric._check_all_processes_locksc Cstj|j|jd|jd}t|}z|j|jdWn.t k rht d|d|j ddYn X| tj|j|jd|jd}t |}z|j|jdWn.t k rt d|d|j d dYn X| dS) Nrz -0.arrow.lockrrrr -rdv.lockzCouldn't acquire lock on r)rirjr8rnrLr`rr(r'rrhrar*r)rrrZlock_file_namer|r%r%r&_check_rendez_vous]s$   zMetric._check_rendez_vouscCs|jdk r|jd|_|jdk r8|jdkr8|j|jrht|jt|j dd}t |j |_nv|jdkr|\}}z4tdt|j dd}t f|dd|D|_Wntk rtddYnX||_||_dS) zClose all the writing process and load/gather the data from all the nodes if main node or all_process is True. NrrO)rjrEcSsg|] }d|iqS)filenamer%)r5fr%r%r&rsz$Metric._finalize..zError in finalize: another metric instance is already using the local cache file. Please specify an experiment_id to avoid collision between distributed metric instances.)rxfinalizerrar*r^r rnrrOr Z from_bufferrwgetvaluerzrZ read_filesFileNotFoundErrorrhr}r~)rreaderr}r~r%r%r& _finalizeqs*      zMetric._finalize predictions referencesc s||d|dkr>|dkr>fddjD}|n2fddjD}|rptd|dtjfd djD}fd dD}td d |Drjf|d_d_ j d krj j j jdfddjD}tjjf||}W5QRXjdk rDd_` d_ nXtttjjD]@\} } t d| ` d_ `d_t| | qZ|SdSdS)aiCompute the metrics. Usage of positional arguments is not allowed to prevent mistakes. Args: predictions (list/array/tensor, optional): Predictions. references (list/array/tensor, optional): References. **kwargs (optional): Keyword arguments that will be forwarded to the metrics :meth:`_compute` method (see details in the docstring). Return: dict or None - Dictionary with the metrics if this metric is run on the main process (``process_id == 0``). - None if the metric is not run on the main process (``process_id != 0``). Example: ```py >>> from datasets import load_metric >>> metric = load_metric("accuracy") >>> accuracy = metric.compute(predictions=model_prediction, references=labels) ``` rNcsi|]}|kr|dqSrr%r5k all_kwargsr%r& sz"Metric.compute..csg|]}|kr|qSr%r%rrr%r&rsz"Metric.compute..zMetric inputs are missing: . All required inputs are csi|]}||qSr%r%r5Z input_namerr%r&rscs i|]}|jkr||qSr%rr)r"rr%r&rs css|]}|dk VqdSrr%)r5vr%r%r&r7sz!Metric.compute..r)r?csi|]}|j|qSr%)rzrr+r%r&rsz Removing )rOupdaterhr@anyvaluesrtrr{rrarzZ set_formatrErVrrb_computerwreversedzipr~r}loggerrxriremover*) rrrr"Zmissing_kwargsZmissing_inputsinputsZcompute_kwargsoutputrrr%)rr"rr&rssF       zMetric.computec s|fdd|D}|r0td|dtj||d|fddjDjjjdkrrzjWntj k rvt fd d Drt t fd dDd }d dt d|dt |d }nftjddgkrHdjd}dfdd jD}||7}n djdt|dt|}t|dYnXdS)aAdd a batch of predictions and references for the metric's stack. Args: predictions (list/array/tensor, optional): Predictions. references (list/array/tensor, optional): References. Example: ```py >>> from datasets import load_metric >>> metric = load_metric("accuracy") >>> metric.add_batch(predictions=model_prediction, references=labels) ``` csg|]}|jkr|qSr%rrr+r%r&rs z$Metric.add_batch..Bad inputs for metric: rrcsi|]}||qSr%r%r5Z intput_namebatchr%r&rsz$Metric.add_batch..Nc3s.|]&}t|tttkVqdSr)rAnextiterrr5crr%r&r7sz#Metric.add_batch..cs(g|] }t|tkr|qSr%)rAr)rcol0r%r&rsrzMismatch in the number of z (z) and rrr@Metric inputs don't match the expected format. Expected format: , c3s&|]}d|dt|VqdSzInput z: NrCrrr%r&r7szPPredictions and/or references don't match the expected format. Expected format: z, Input predictions: z, Input references: )rhr@rOrEZ encode_batchrx _init_writerZ write_batchpa ArrowInvalidrrrrAsortedr8rC)rrrr" bad_inputsZbad_col error_msgerror_msg_inputsr%)rrrr&rts2  , zMetric.add_batch) prediction referencec sfdd|D}|r0td|dtj||d|fddjDjjjdkrrzjWnPtj k rd jd }d fd d jD}||7}t|dYnXdS) aAdd one prediction and reference for the metric's stack. Args: prediction (list/array/tensor, optional): Predictions. reference (list/array/tensor, optional): References. Example: ```py >>> from datasets import load_metric >>> metric = load_metric("accuracy") >>> metric.add(predictions=model_predictions, references=labels) ``` csg|]}|jkr|qSr%rrr+r%r&rs zMetric.add..rrrcsi|]}||qSr%r%rexampler%r&rszMetric.add..Nrrc3s&|]}d|dt|VqdSrrrrr%r&r7$szMetric.add..) rhr@rOrEZencode_examplerxrwriterrr8)rrrr"rrrr%)rrr&ru s" z Metric.addc Cs,|jdkr|jdkrtj|j|jd|jd}t||_z|jj |dWn.t k r~t d|d|jddYnX|j rt |_t|jj|j|jd |_nJd|_|jdks|jdkr|\}}||_||_t|jj|j|jd |_|jdkr(|jdkr ||jn|dS) NrrrrrzXError in _init_writer: another metric instance is already using the local cache file at rr)rOstreamry)rOrjry)r`rarirjr8rnrLrr|r( TimeoutErrorrhr^rZBufferOutputStreamrwr rErOryrxr{rrrr*r)rr'rr{rr%r%r&r*sD          zMetric._init_writercCstdS)aConstruct the MetricInfo object. See `MetricInfo` for details. Warning: This function is only called once and the result is cached for all following .info() calls. Returns: info: (MetricInfo) The metrics information NNotImplementedErrorr+r%r%r&reQs z Metric._info)download_config dl_managercCsN|dkr@|dkr.t}tj|jd|_d|_t|j||jd}| |dS)aDownloads and prepares dataset for reading. Args: download_config (:class:`DownloadConfig`, optional): Specific download configuration parameters. dl_manager (:class:`DownloadManager`, optional): Specific download manager to use. NZ downloadsF)Z dataset_namerrn) r rirjr8rnr_Zforce_downloadrrK_download_and_prepare)rrrr%r%r&download_and_prepare\s zMetric.download_and_preparecCsdS)aYDownloads and prepares resources for the metric. This is the internal implementation to overwrite called when user calls `download_and_prepare`. It should download all required resources for the metric. Args: dl_manager (:class:`DownloadManager`): `DownloadManager` used to download and cache data. Nr%)rrr%r%r&rss zMetric._download_and_preparecKstdS)zEThis method defines the common API for all the metrics in the libraryNr)rrrr"r%r%r&r~szMetric._computecCs\t|dr|jdk r|jt|dr<|jdk r<|jt|drJ|`t|drX|`dS)Nrr|rxrz)hasattrrr*r|rxrzr+r%r%r&__del__s    zMetric.__del__) NFNrrNNr[r\)r)r)NN)&r.r/r0r1rrrXrYrgrfloatrrrrmrrrrrrrrdictrsrtrurrrer rrrrrrrr%r%r%r&rZsX D !#"G/! '   rZ)3r1rirqrtypingrrrrrrnumpyroZpyarrowrrr Z arrow_datasetr Z arrow_readerr Z arrow_writerr Zdownload.download_configr Zdownload.download_managerrrOrrErrZnamingrZutils.deprecation_utilsrZutils.filelockrrrZ utils.loggingrZutils.py_utilsrrr.rrrCrDrZr%r%r%r&s0            E