U -eF @sdZddlZddlZddlmZmZddlmZddl m Z ddl m Z m Z mZddlmZddlmZdd lmZdd lmZmZdd lmZmZdd lmZdd lmZddlm Z m!Z!dZ"eddd\Z#Z$ee#e$dd\Z#Z$e%e#Z#ddddgZ&dhdde'DBZ(ej)*deddZ+dd Z,ej)*d!ej-ej.gd"d#Z/d$d%Z0ej)*d&e&ej)*d'ed(d)Z1d*d+Z2ej)*d,d-d.d/Z3d0d1Z4d2d3Z5d4d5Z6d6d7Z7d8d9Z8ej)*d:d;dZ9d?d@Z:ej)*dAe&dBdCZ;dDdEZdJej?gej?dJggfd'dIidJdKgdKdJggfidJdKgddLggfgdMdNZ@dOdPZAdQdRZBdSdTZCdUdVZDej)*dWdXdYgej)*dZdd[gd\d]ZEd^d_ZFdS)`zF Tests for HDBSCAN clustering algorithm Based on the DBSCAN test code N)sparsestats)distance)HDBSCAN)CONDENSED_dtype_condense_tree _do_labelling)_OUTLIER_ENCODING) make_blobs)fowlkes_mallows_score)_VALID_METRICSeuclidean_distances)BallTreeKDTree)StandardScaler)shuffle)assert_allcloseassert_array_equal ) n_samples random_state)rkdtreeballtreebruteautocCsh|]\}}|dqS)label).0_outr r c/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/sklearn/cluster/tests/test_hdbscan.py %sr% outlier_typec Cstjtjd|}ddddd|}t|d}t|d}t}|dg|d<||g|d <t|}|j|k \}t |dd g||j | \}t |dd gt t dd t t d d } t|| } t | j|j| d S) O Tests if np.inf and np.nan data are each treated as special outliers. )infinitemissingcSs||kSNr xyr r r$2z#test_outlier_data..cSs t|Sr*)npisnanr+r r r$r.3r/rprobrrN)r0infnanr Xcopyrfitlabels_ZnonzerorZprobabilities_listrange) r&ZoutlierZ prob_checkrr2 X_outliermodelmissing_labels_idxZmissing_probs_idxZ clean_indices clean_modelr r r$test_outlier_data(s.     rBc Cstt}|}tddd|}t||tt|t}|t ksHt t t |}|dks^t d}t jt|dtdddtW5QRXd}d|d <d |d <t jt|dtdd |W5QRXd S)zy Tests that HDBSCAN works with precomputed distance matrices, and throws the appropriate errors when needed. precomputedT)metricr9\(\?z*The precomputed distance matrix.*has shapematchz'The precomputed distance matrix.*valuesr)rr3r3)r3rrDN)r r8r9r fit_predictrlenset OUTLIER_SETn_clusters_trueAssertionErrorr r-pytestraises ValueError)DZ D_originallabels n_clustersscoremsgr r r$test_hdbscan_distance_matrixHs     rWsparse_constructorcCszttt}|t|}t|d}d|||k<||}| t dd |}t t |t}|tksvtdS)zA Tests that HDBSCAN works with sparse distance matrices. 2rCrHN)rZ squareformZpdistr8r0maxrZscoreatpercentileflattenZeliminate_zerosrrIrJrKrLrMrN)rXrR thresholdrSrTr r r$#test_hdbscan_sparse_distance_matrixfs r^cCsBtt}tt|t}|tks(ttt |}|dks>tdS)z Tests that HDBSCAN works with feature array, including an arbitrary goodness of fit check. Note that the check is a simple heuristic. rEN) rrIr8rJrKrLrMrNr r-)rSrTrUr r r$test_hdbscan_feature_arrayys    r_algorDc Cst|dt}tt|t}|tks,t|dkr8dStt d}dt tj didt tj diddidt tj dd d |d}t|||d }|||jkrtt|tW5QRXn4|d krtt|tW5QRXn |tdS) z Tests that HDBSCAN works with the expected combinations of algorithms and metrics, or raises the expected errors. ) algorithm)rrN)rrVr3p)rcw)Z mahalanobis seuclideanZ minkowski wminkowskirarD metric_paramsrg)rrIr8rJrKrLrMrNrrr0eyeshapeonesget valid_metricsrOrPrQr:Zwarns FutureWarning)r`rDrSrTZ ALGOS_TREESrihdbr r r$test_hdbscan_algorithmss8   rqcCs6tt}|d}tt|t}|tks2tdS)z Tests that HDBSCAN can generate a sufficiently accurate dbscan clustering. This test is more of a sanity check than a rigorous evaluation. TODO: Improve and strengthen this test if at all possible. 333333?N) rr:r8dbscan_clusteringrJrKrLrMrN)Z clustererrSrTr r r$test_dbscan_clusterings  rt cut_distance)皙??r3c Cstdd}tdd}t}tjdg|d<dtjg|d<tjtjg|d<t|}|j|d}t ||k}t |ddgt ||k}t |dgt t t d t ||}t||} | j|d} t | ||d S) r'r)rr(r3rrdr4)rurN)r r8r9r0r6r7rr:rsZ flatnonzerorr<rKr=) ruZ missing_labelZinfinite_labelr>r?rSr@Zinfinite_labels_idxZ clean_idxrAZ clean_labelsr r r$#test_dbscan_clustering_outlier_datas       rxcCsbtdddd\}}t|}tdddt|jdid |}tt |t }|t ks^t d S) zQ Tests that HDBSCAN using `BallTree` works with higher-dimensional data. rYr@)rrZ n_featuresrrfrbr3rhN) r r fit_transformrr0rlrkrIrJrKrLrMrN)Hr-rSrTr r r$test_hdbscan_high_dimensionals r|cCsBtddttjdidt}tt|t}|t ks>t dS)z4 Tests that HDBSCAN using `BallTree` works. rfrbr3)rDriN) rr0rlr8rkrIrJrKrLrMrNrSrTr r r$!test_hdbscan_best_balltree_metricsr~cCs8tttddt}tt|t}|dks4tdS)z Tests that HDBSCAN correctly does not generate a valid cluster when the `min_cluster_size` is too large for the data. r3min_cluster_sizerN)rrJr8rIrKrLrNr}r r r$test_hdbscan_no_clusterssrcCs\tdttdD]F}t|dt}dd|D}t|dkrtt||kstqdS)zb Test that the smallest non-noise cluster has at least `min_cluster_size` many points rdr3rcSsg|]}|dkr|qS)rr )r!rr r r$ sz1test_hdbscan_min_cluster_size..rN) r=rJr8rrIr0minZbincountrN)rrSZ true_labelsr r r$test_hdbscan_min_cluster_sizes  rcCs6tj}t|dt}tt|t}|tks2t dS)zA Tests that HDBSCAN works when passed a callable metric. rHN) r euclideanrrIr8rJrKrLrMrN)rDrSrTr r r$test_hdbscan_callable_metric srtreekdZballc Cs8tdd|dd}tt|tW5QRXdS)z Tests that HDBSCAN correctly raises an error when passing precomputed data while requesting a tree-based algorithm. rCZprims_rrDraN)rrOrPrQr:r8)rrpr r r$"test_hdbscan_precomputed_non_brutes rc Cs ttj}tt|t}|dks*tt t}| }t|j}t ||t j dft jdffD]\}}t }||d<t|j}tt|t}|dkst|dt|dkst| }||d<t|j}t ||qhd}tjt|dtd d d |W5QRXd S) z Tests that HDBSCAN works correctly when passing sparse feature data. Evaluates correctness by comparing against the same data passed as a dense array. rr(r)rrrrz4Sparse data matrices only support algorithm `brute`.rFrrrN)rr:r8r;rJrKrLrNr csr_matrixr9rr0r6r7r rOrPrQ) Z dense_labelsrTZ _X_sparseZX_sparseZ sparse_labelsZ outlier_valr&ZX_denserVr r r$test_hdbscan_sparses*     rracCsddg}tdd|dd\}}tdd|}t||j|jD]*\}}}t||d d d t||d d d q.zV.* is not a valid metric for a .*-based algorithm\. Please select a different metric\.rFr)rarDrrN) rOrPrQrr:r8r<rKrrnrrJ)Zmetric_callablerVZmetrics_not_kdr r r$ test_hdbscan_tree_invalid_metrics rc Cs>tttdd}d}tjt|d|tW5QRXdS)zx Tests that HDBSCAN correctly raises an error when setting `min_samples` larger than the number of samples. r3)rz min_samples (.*) must be at mostrFN)rrJr8rOrPrQr:)rprVr r r$!test_hdbscan_too_many_min_samplessrc CsHt}tj|d<d}tdd}tjt|d||W5QRXdS)zu Tests that HDBSCAN correctly raises an error when providing precomputed distances with `np.nan` values. rz(np.nan values found in precomputed-denserCrHrFN) r8r9r0r7rrOrPrQr:)ZX_nanrVrpr r r$"test_hdbscan_precomputed_dense_nans   rrTFepsilonrvc sd}t||ddgddgddggd\}t|}t|j|jd}|d|d|dh}|dd|dd |ddi}t|||||d fd d ttDfd d ttD} t | j } t | dS)zR Tests that the `_do_labelling` helper function correctly assigns labels. 0rr)rrrrdrrr3condensed_treeclusterscluster_label_maprrcs$i|]}|t|kddqS)r)r0wherer!Z_y)r-r r$ sz+test_labelling_distinct..csi|]}||qSr r r)first_with_labelrSr r$rsN) r rr:rZ_single_linkage_tree_rrr<rKr0Z vectorizermr) Zglobal_random_seedrrrr8ZestrrrZ y_to_labelsZaligned_targetr )rrSr-r$test_labelling_distincts6 rcCsd}d}tjdd|dfddd|dfddgtd }t||h|d|ddid dd }|d dk}t|t|d kksttt||h|d|ddid dd }|d |k}t|t|d kkstdS)z Tests that the `_do_labelling` helper function correctly thresholds the incoming lambda values given various `cluster_selection_epsilon` values. r4g?rdr3)r4r3rvr3r)r4rrr3)r4rrrr3)ZdtypeTrvaluerN)r0arrayrrsumrN)rZ MAX_LAMBDArrSZ num_noiser r r$test_labelling_thresholdings:     r)G__doc__numpyr0rOZscipyrrZ scipy.spatialrZsklearn.clusterrZsklearn.cluster._hdbscan._treerrrZ sklearn.cluster._hdbscan.hdbscanr Zsklearn.datasetsr Zsklearn.metricsr Zsklearn.metrics.pairwiser r Zsklearn.neighborsrrZsklearn.preprocessingrZ sklearn.utilsrZsklearn.utils._testingrrrMr8r-rzZ ALGORITHMSitemsrLmarkZ parametrizerBrWrZ csc_matrixr^r_rqrtrxr|r~rrrrrrrrrr6rrrrrrrr r r r$s              (       $  !      $