Curvature as a tool for evaluating dimensionality reduction and estimating intrinsic dimension

2025-09-16

AI Summary

This paper addresses the challenges of evaluating dimensionality reduction (DR) effectiveness and estimating intrinsic data dimensionality. We propose a geometric profiling method based on sectional curvature in discrete metric spaces, which characterizes large-scale data geometry via metric relationships among point triplets. For the first time, this approach systematically introduces differential-geometric curvature into quantitative DR quality assessment and intrinsic dimension estimation—without requiring embedded coordinates or manifold assumptions, thus ensuring both theoretical rigor and computational feasibility. Experiments across diverse synthetic and real-world datasets demonstrate that our method robustly discriminates DR algorithm performance, achieves significantly lower intrinsic dimension estimation error than state-of-the-art methods (e.g., MDS- and PCA-based estimators), and successfully uncovers latent negative curvature in empirical networks—including social and biological networks.

Technology Category

• Machine Learning: Learning with Manifolds
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Data Mining & Knowledge Management: Anomaly/Outlier Detection

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Utilizing recently developed abstract notions of sectional curvature, we introduce a method for constructing a curvature-based geometric profile of discrete metric spaces. The curvature concept that we use here captures the metric relations between triples of points and other points. More significantly, based on this curvature profile, we introduce a quantitative measure to evaluate the effectiveness of data representations, such as those produced by dimensionality reduction techniques. Furthermore, Our experiments demonstrate that this curvature-based analysis can be employed to estimate the intrinsic dimensionality of datasets. We use this to explore the large-scale geometry of empirical networks and to evaluate the effectiveness of dimensionality reduction techniques.

Problem

• Evaluating dimensionality reduction effectiveness via curvature profiles
• Estimating intrinsic dimensionality of datasets using curvature analysis
• Exploring large-scale geometry of networks with curvature-based methods

Innovation

• Curvature-based geometric profile construction
• Evaluating dimensionality reduction effectiveness quantitatively
• Estimating intrinsic dataset dimensionality via curvature