MCGI: Manifold-Consistent Graph Indexing for Billion-Scale Disk-Resident Vector Search

2026-01-05

arXiv.org

AI Summary

This work addresses the performance degradation of disk-based high-dimensional graph indexing caused by the Euclidean–geodesic mismatch. To mitigate this issue, the authors propose a geometry-aware approximate nearest neighbor search method that integrates local intrinsic dimensionality (LID) into both graph construction and dynamic beam search. This approach adaptively aligns with the underlying data manifold without requiring any preset hyperparameters. The method substantially enhances topological connectivity and retrieval accuracy for high-dimensional data: on GIST1M, it achieves 5.8× higher throughput than DiskANN at 95% recall, and on the billion-scale SIFT1B dataset, it reduces query latency by 3× under high-recall conditions, while maintaining comparable performance on low-dimensional data.

Technology Category

• Search and Optimization: Distributed Search
• Machine Learning: Graph-based Machine Learning
• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community

Application Category

• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Graph-based Approximate Nearest Neighbor (ANN) search often suffers from performance degradation in high-dimensional spaces due to the ``Euclidean-Geodesic mismatch,''where greedy routing diverges from the underlying data manifold. To address this, we propose Manifold-Consistent Graph Indexing (MCGI), a geometry-aware and disk-resident indexing method that leverages Local Intrinsic Dimensionality (LID) to dynamically adapt search strategies to the data's intrinsic geometry. Unlike standard algorithms that treat dimensions uniformly, MCGI modulates its beam search budget based on in situ geometric analysis, eliminating dependency on static hyperparameters. Theoretical analysis confirms that MCGI enables improved approximation guarantees by preserving manifold-consistent topological connectivity. Empirically, MCGI achieves 5.8$\times$ higher throughput at 95\% recall on high-dimensional GIST1M compared to state-of-the-art DiskANN. On the billion-scale SIFT1B dataset, MCGI further validates its scalability by reducing high-recall query latency by 3$\times$, while maintaining performance parity on standard lower-dimensional datasets.

Problem

• Approximate Nearest Neighbor
• Euclidean-Geodesic mismatch
• high-dimensional search
• manifold inconsistency
• graph-based indexing

Innovation

• Manifold-Consistent Graph Indexing
• Local Intrinsic Dimensionality
• Disk-Resident ANN Search
• Euclidean-Geodesic Mismatch
• Adaptive Beam Search