o ?HhK @sDdZddlZddlZddlmZddlmZddlm Z ddl m Z m Z m Z ddlmZddlmZdd lmZdd lmZmZdd lmZmZdd lmZdd lmZddlmZm Z ddl!m"Z"m#Z#eddd\Z$Z%ee$e%dd\Z$Z%e&e$Z$gdZ'dhdde(DBZ)doddZ*ej+,deddZ-dd Z.ej+,d!ge#e"d"d#Z/d$d%Z0ej+,d&e'ej+,d'ed(d)Z1d*d+Z2ej+,d,d-d.d/Z3d0d1Z4d2d3Z5d4d5Z6d6d7Z7ej+,d8d9d:gd;d<Z8ej+,d=e#d>d?Z9ej+,d@e'dAdBZ:dCdDZ;dEdFZgej>dIggfd'dHidIdJgdJdIggfidIdJgdKdLggfgdMdNZ?ej+,d=e#dOdPZ@ej+,d=e#dQdRZAdSdTZBdUdVZCdWdXZDej+,dYdZd[gej+,d\dd]gd^d_ZEd`daZFej+,dbdcddgdedfZGej+,dgdhdigdjdkZHej+,dld9d:gdmdnZIdS)pzF Tests for HDBSCAN clustering algorithm Based on the DBSCAN test code N)stats)distance)HDBSCAN)CONDENSED_dtype_condense_tree _do_labelling)_OUTLIER_ENCODING) make_blobs)fowlkes_mallows_score)_VALID_METRICSeuclidean_distances)BallTreeKDTree)StandardScaler)shuffle)assert_allcloseassert_array_equal)CSC_CONTAINERSCSR_CONTAINERS ) n_samples random_state)r)kd_tree ball_treebruteautocCsh|]\}}|dqS)label).0_outr r b/home/air/sanwanet/gpt-api/venv/lib/python3.10/site-packages/sklearn/cluster/tests/test_hdbscan.py &r%Gz?cCs2tt|t}|dksJt|t|ksJdS)N)lenset OUTLIER_SETr y)labels threshold n_clustersr r r$check_label_quality)s r0 outlier_typec Cstjtjd|}ddddd|}t|d}t|d}t}|dg|d<||g|d <t|}|j|k \}t |dd g||j | \}t |dd gt t dd t t d d } t|| } t | j|j| d S) O Tests if np.inf and np.nan data are each treated as special outliers. )infinitemissingcSs||kSNr xr,r r r$9sz#test_outlier_data..cSs t|Sr5)npisnanr6r r r$r8:s rprobrrN)r9infnanrXcopyrfitlabels_nonzerorprobabilities_listrange) r1outlier prob_checkrr; X_outliermodelmissing_labels_idxmissing_probs_idx clean_indices clean_modelr r r$test_outlier_data/s.     rQcCstt}|}tddd|}t||t|d}tjt |dtdddtWdn1s7wYd}d|d <d |d <tjt |dtdd |WddS1sawYdS) zy Tests that HDBSCAN works with precomputed distance matrices, and throws the appropriate errors when needed. precomputedT)metricrBz*The precomputed distance matrix.*has shapematchNz'The precomputed distance matrix.*valuesr)rr<r<)r<rrS) r rArBr fit_predictrr0pytestraises ValueError)D D_originalr-msgr r r$test_hdbscan_distance_matrixOs "r^sparse_constructorcCsfttt}|t|}t|d}d|||k<||}| t dd |}t |dS)zA Tests that HDBSCAN works with sparse distance matrices. 2rRrVN) r squareformpdistrAr9maxrscoreatpercentileflatteneliminate_zerosrrWr0)r_r[r.r-r r r$#test_hdbscan_sparse_distance_matrixgs  rhcCstt}t|dS)z Tests that HDBSCAN works with feature array, including an arbitrary goodness of fit check. Note that the check is a simple heuristic. N)rrWrAr0r-r r r$test_hdbscan_feature_arrayys  rjalgorScCs t|dt}t||dvrdSttd}dttjdidt tjdiddidt tjdd d  |d}t|||d }|||j vrht t|tWddS1sawYdS|d krt t|tWddS1swYdS|tdS) z Tests that HDBSCAN works with the expected combinations of algorithms and metrics, or raises the expected errors. ) algorithm)rrN)rrVr<p)rnw) mahalanobis seuclidean minkowski wminkowski)rlrS metric_paramsrt)rrWrAr0rr r9eyeshapeonesget valid_metricsrXrYrZrCwarns FutureWarning)rkrSr- ALGOS_TREESruhdbr r r$test_hdbscan_algorithmss8  "  "rcCs&tt}|d}t|dddS)z Tests that HDBSCAN can generate a sufficiently accurate dbscan clustering. This test is more of a sanity check than a rigorous evaluation. 333333?gq= ףp?)r.N)rrCrAdbscan_clusteringr0) clustererr-r r r$test_dbscan_clusterings  r cut_distance)皙??r<c Cstdd}tdd}t}tjdg|d<dtjg|d<tjtjg|d<t|}|j|d}t ||k}t |ddgt ||k}t |dgt t t d t ||}t||} | j|d} t | ||d S) r2r4rr3r<rror=)rrN)rrArBr9r?r@rrCr flatnonzerorrGr*rH) r missing_labelinfinite_labelrKrLr-rMinfinite_labels_idx clean_idxrP clean_labelsr r r$#test_dbscan_clustering_outlier_datas      rcCs.tddttjdidt}t|dS)z4 Tests that HDBSCAN using `BallTree` works. rrrmr<)rSruN)rr9rxrArwrWr0rir r r$!test_hdbscan_best_balltree_metrics  rcCs.tttddt}t|tsJdS)z Tests that HDBSCAN correctly does not generate a valid cluster when the `min_cluster_size` is too large for the data. r<min_cluster_sizeN)rr)rArWr*issubsetr+rir r r$test_hdbscan_no_clusterssrcCs\tdttdD]#}t|dt}dd|D}t|dkr+tt||ks+JqdS)zb Test that the smallest non-noise cluster has at least `min_cluster_size` many points ror<rcSsg|]}|dkr|qS)rr )r!rr r r$ r&z1test_hdbscan_min_cluster_size..rN)rHr)rArrWr9minbincount)rr- true_labelsr r r$test_hdbscan_min_cluster_sizes rcCs"tj}t|dt}t|dS)zA Tests that HDBSCAN works when passed a callable metric. rVN)r euclideanrrWrAr0)rSr-r r r$test_hdbscan_callable_metrics rtreerrcCsNtd|d}d}tjt|d|tWddS1s wYdS)z Tests that HDBSCAN correctly raises an error when passing precomputed data while requesting a tree-based algorithm. rRrSrlz%precomputed is not a valid metric forrTN)rrXrYrZrCrA)rr~r]r r r$"test_hdbscan_precomputed_non_brutes  "r csr_containerc Cs ttj}t||t}|}t|j}t||tjdftj dffD]7\}}t}||d<t|j}t||dt |dksLJ|}||d<t|j}t||q)d}t j t |dtdd d |Wd d S1swYd S) z Tests that HDBSCAN works correctly when passing sparse feature data. Evaluates correctness by comparing against the same data passed as a dense array. r3r4rrrrz4Sparse data matrices only support algorithm `brute`.rTrrrN)rrCrArDr0rBrr9r?r@rrXrYrZ) r dense_labels _X_sparseX_sparse sparse_labels outlier_valr1X_denser]r r r$test_hdbscan_sparse s(  "rrlcCsddg}tdd|dd\}}tdd|}t||j|jD]\}}}t||d d d t||d d d qt|dtjdd t}|jjddksKJ|jjddksUJd S)zj Tests that HDBSCAN centers are calculated and stored properly, and are accurate to the data. )rara)@rirr)rrcenters cluster_stdboth) store_centersr<g?)rtolatol)rlrrN) r rrCzip centroids_medoids_rrArw)rlrHr"r~centercentroidmedoidr r r$test_hdbscan_centers-s rcCstjd}|dd}tddddd|}tj|dd \}}t|dks(J||d kd ks2Jtdd ddd d|}tj|dd \}}t|dksOJ||d kdksYJdS)zS Tests that HDBSCAN single-cluster selection with epsilon works correctly. rror=raeomT)rcluster_selection_epsiloncluster_selection_methodallow_single_cluster) return_countsrg ףp= ?r)rrrrrlN)r9random RandomStaterandrrWuniquer))rng no_structurer- unique_labelscountsr r r$.test_hdbscan_allow_single_cluster_with_epsilonCs2  rcCsxddgddgddgddgg}td|gddd\}}t|j}tt|td |v}|d ks3Jt||d kd S) z Validate that HDBSCAN can properly cluster this difficult synthetic dataset. Note that DBSCAN fails on this (see HDBSCAN plotting example) g333333g333333?r(i)皙?gffffff?皙?rr)rrrrrr'N)r rrCrDr)r*intr )rrAr,r-r/r r r$test_hdbscan_better_than_dbscands  rz kwargs, XrRr<ror(rcCstdddi||dS)zo Tests that HDBSCAN works correctly for array-likes and precomputed inputs with non-finite points. min_samplesr<Nr )rrC)rAkwargsr r r$test_hdbscan_usable_inputsxs rcCsV|td}d}tjt|dtdd|WddS1s$wYdS)zd Tests that HDBSCAN raises the correct error when there are too few non-zero distances. )rrz#There exists points with fewer thanrTrRrVN)r9zerosrXrYrZrrCrrAr]r r r$-test_hdbscan_sparse_distances_too_few_nonzeros "rcCstd}d|ddddf<d|ddddf<||j}||}d}tjt|dtdd |WddS1s?wYdS) zu Tests that HDBSCAN raises the correct error when the distance matrix has multiple connected components. )rr<Nr=z2HDBSCAN cannot be perfomed on a disconnected graphrTrRrV)r9rTrXrYrZrrCrr r r$0test_hdbscan_sparse_distances_disconnected_graphs  "rcCsdd}d}tjt|dtd|dtWdn1s!wYtjt|dtd|dtWdn1sAwYtttj tt j }t |d kr}tjt|dtd|d dtWddS1svwYdSdS) zR Tests that HDBSCAN correctly raises an error for invalid metric choices. cSs|Sr5r )r7r r r$r8sz2test_hdbscan_tree_invalid_metric..zV.* is not a valid metric for a .*-based algorithm\. Please select a different metric\.rTr)rlrSNrr) rXrYrZrrCrArGr*r rzrr))metric_callabler]metrics_not_kdr r r$ test_hdbscan_tree_invalid_metrics "rcCsTtttdd}d}tjt|d|tWddS1s#wYdS)zx Tests that HDBSCAN correctly raises an error when setting `min_samples` larger than the number of samples. r<)rz min_samples (.*) must be at mostrTN)rr)rArXrYrZrC)r~r]r r r$!test_hdbscan_too_many_min_sampless  "rcCs^t}tj|d<d}tdd}tjt|d||WddS1s(wYdS)zu Tests that HDBSCAN correctly raises an error when providing precomputed distances with `np.nan` values. rz(np.nan values found in precomputed-denserRrVrTN) rArBr9r@rrXrYrZrC)X_nanr]r~r r r$"test_hdbscan_precomputed_dense_nans   "rrTFepsilonrc sd}t||ddgddgddggd\}t|}t|j|jd}|d|d|dh}|dd|dd |ddi}t|||||d fd d ttDfd d ttD} t | j } t | dS)zR Tests that the `_do_labelling` helper function correctly assigns labels. 0rr)rrrror(rr<condensed_treeclusterscluster_label_maprrcs$i|]}|t|kddqS)r)r9wherer!_y)r,r r$ s$z+test_labelling_distinct..csi|] }||qSr r r)first_with_labelr-r r$rsN) r rrCr_single_linkage_tree_rrrGr*r9 vectorizeryr) global_random_seedrrrrAestrrr y_to_labelsaligned_targetr )rr-r,r$test_labelling_distincts4 rcCsd}d}tjdd|dfddd|dfddgtd }t||h|d|ddid dd }|d dk}t|t|d kks:Jt||h|d|ddid dd }|d |k}t|t|d kks\JdS)z Tests that the `_do_labelling` helper function correctly thresholds the incoming lambda values given various `cluster_selection_epsilon` values. r=g?ror<)r=r<rr<r)r=r(rr<)r=rrr<)dtypeTrvaluerN)r9arrayrrsum)r MAX_LAMBDArr- num_noiser r r$test_labelling_thresholdings:     rrrrcCshtjd}|d}t|}d}tjt|dtd|d|WddS1s-wYdS)zCheck that we raise an error if the centers are requested together with a precomputed input matrix. Non-regression test for: https://github.com/scikit-learn/scikit-learn/issues/27893 r)droz>Cannot store centers when using a precomputed distance matrix.rTrR)rSrN) r9rrr rXrYrZrrC)rrrAX_disterr_msgr r r$0test_hdbscan_error_precomputed_and_store_centers%s  "r valid_algorrcCstd|dtdS)zTest that HDBSCAN works with the "cosine" metric when the algorithm is set to "brute" or "auto". Non-regression test for issue #28631 cosinerN)rrWrA)rr r r$*test_hdbscan_cosine_metric_valid_algorithm5sr invalid_algocCsJtd|d}tjtdd|tWddS1swYdS)zTest that HDBSCAN raises an informative error is raised when an unsupported algorithm is used with the "cosine" metric. rrzcosine is not a valid metricrTN)rrXrYrZrWrA)rhdbscanr r r$,test_hdbscan_cosine_metric_invalid_algorithm?s  "r)r')J__doc__numpyr9rXscipyr scipy.spatialrsklearn.clusterrsklearn.cluster._hdbscan._treerrr sklearn.cluster._hdbscan.hdbscanrsklearn.datasetsr sklearn.metricsr sklearn.metrics.pairwiser r sklearn.neighborsr rsklearn.preprocessingr sklearn.utilsrsklearn.utils._testingrrsklearn.utils.fixesrrrAr, fit_transform ALGORITHMSitemsr+r0mark parametrizerQr^rhrjrrrrrrrrrrrrrr?rrrrrrrrrrrr r r r$s               '       " !        $)