SHINE: A Scalable HNSW Index in Disaggregated Memory

2025-07-23

AI Summary

To address two key challenges in approximate nearest neighbor (ANN) search for high-dimensional vectors on large-scale disaggregated memory architectures—accuracy degradation due to graph partitioning and network bandwidth bottlenecks from remote memory access—this paper proposes DistHNSW, a graph-preserving distributed HNSW indexing framework. Methodologically, it introduces (1) a lossless graph partitioning and reconstruction mechanism that guarantees retrieval accuracy identical to single-node HNSW, and (2) a novel cross-node logical cache coordination scheme leveraging fine-grained vector caching and RDMA-accelerated cache aggregation to significantly improve cache hit rates and bandwidth utilization. Evaluated on billion-scale high-dimensional datasets, DistHNSW achieves Recall@10 comparable to the single-node baseline, delivers a 2.3× throughput improvement, enhances cache efficiency by 3.8×, and demonstrates strong horizontal scalability.

Technology Category

• Search and Optimization: Distributed Search
• Machine Learning: Graph-based Machine Learning
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems

Application Category

• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

Approximate nearest neighbor (ANN) search is a fundamental problem in computer science for which in-memory graph-based methods, such as Hierarchical Navigable Small World (HNSW), perform exceptionally well. To scale beyond billions of high-dimensional vectors, the index must be distributed. The disaggregated memory architecture physically separates compute and memory into two distinct hardware units and has become popular in modern data centers. Both units are connected via RDMA networks that allow compute nodes to directly access remote memory and perform all the computations, posing unique challenges for disaggregated indexes. In this work, we propose a scalable HNSW index for ANN search in disaggregated memory. In contrast to existing distributed approaches, which partition the graph at the cost of accuracy, our method builds a graph-preserving index that reaches the same accuracy as a single-machine HNSW. Continuously fetching high-dimensional vector data from remote memory leads to severe network bandwidth limitations, which we overcome by employing an efficient caching mechanism. Since answering a single query involves processing numerous unique graph nodes, caching alone is not sufficient to achieve high scalability. We logically combine the caches of the compute nodes to increase the overall cache effectiveness and confirm the efficiency and scalability of our method in our evaluation.

Problem

• Scalable ANN search in disaggregated memory architecture
• Maintaining accuracy in distributed graph-based index
• Overcoming network bandwidth limits with efficient caching

Innovation

• Scalable HNSW index for disaggregated memory
• Graph-preserving index maintains single-machine accuracy
• Combined caching mechanism overcomes bandwidth limitations