Robust high-dimensional non-parametric representative points via density-guided clustering

Abstract

Representative points (RPs) provide a sparse yet informative summary of complex datasets, enabling efficient data compression, approximation, and resampling. Traditional RP generation methods rely on parametric assumptions about the underlying distribution, which limits their applicability to real-world data where the true distribution function is unknown. To address this limitation, we introduce a non-parametric framework that intrinsically integrates kernel density estimation into the clustering process through a density-guided distance. This density-guided distance dynamically balances geometric proximity in the original data space with probabilistic proximity in the estimated density space. Extensive experiments on high-dimensional real-world datasets and synthetic datasets demonstrate that the proposed density-guided non-parametric RPs (DGNPREPs) consistently match or outperform existing parametric RPs in terms of moment accuracy, distribution approximation, density estimation fidelity, and statistical validation. Crucially, hypothesis testing across all considered cases fails to reject the null hypothesis of no difference between estimators based on the proposed DGNPREPs and the true parameters, whereas several existing parametric methods show significant deviations. Quantitatively, DGNPREPs achieve up to a 99% reduction in bias, 78% reduction in mean squared error, 66% improvement in uniformity, 60% reduction in L2-distance, and 50% reduction in confidence interval length. This work provides a robust, accurate, and flexible approach to data summarization that excels when the underlying distribution is unknown while remaining competitive when parametric assumptions happen to hold.