🤖 AI Summary
High-dimensional anomaly detection suffers from the “curse of dimensionality,” rendering conventional methods ineffective; existing approaches often compromise interpretability or computational efficiency. This paper introduces a novel outlierness statistic based on “distance-to-distance,” leveraging the asymptotic concentration of pairwise distances and inner products in high-dimensional spaces—transforming the curse of dimensionality into a discriminative advantage that ensures asymptotic separability between anomalies and inliers. We theoretically establish the existence of a non-vanishing separation boundary for this statistic in high dimensions. Furthermore, we propose a distribution-free random rotation testing framework, requiring no parametric assumptions and exhibiting strong robustness. Experiments on synthetic data and diverse real-world high-dimensional datasets—including gene expression profiles and image features—demonstrate that our method significantly outperforms state-of-the-art baselines: it achieves substantially higher recall while maintaining low false positive rates, combining statistical rigor, computational feasibility, and result interpretability.
📝 Abstract
Reliable outlier detection in high-dimensional data is crucial in modern science, yet it remains a challenging task. Traditional methods often break down in these settings due to their reliance on asymptotic behaviors with respect to sample size under fixed dimension. Furthermore, many modern alternatives introduce sophisticated statistical treatments and computational complexities. To overcome these issues, our approach leverages intuitive geometric properties of high-dimensional space, effectively turning the curse of dimensionality into an advantage. We propose two new outlyingness statistics based on observation's relational patterns with all other points, measured via pairwise distances or inner products. We establish a theoretical foundation for our statistics demonstrating that as the dimension grows, our statistics create a non-vanishing margin that asymptotically separates outliers from non-outliers. Based on this foundation, we develop practical outlier detection procedures, including a simple clustering-based algorithm and a distribution-free test using random rotations. Through simulation experiments and real data applications, we demonstrate that our proposed methods achieve a superior balance between detection power and false positive control, outperforming existing methods and establishing their practical utility in high-dimensional settings.