GEM: A Native Graph-based Index for Multi-Vector Retrieval

2026-03-20

AI Summary

This work addresses the challenge of balancing fine-grained semantic representation and retrieval efficiency in multi-vector retrieval, which has been hindered by the absence of efficient indexing mechanisms. To this end, we propose GEM, a native graph-based indexing framework that, for the first time, designs an index structure specifically for sets of vectors. GEM integrates set-level clustering, local proximity graph connectivity, and global navigation, while decoupling graph construction metrics from relevance scoring. It further introduces semantic shortcuts and a multi-entry beam search mechanism, enhanced with quantized distance estimation, to significantly accelerate retrieval. Experimental results demonstrate that GEM achieves up to a 16× speedup over existing methods across multiple benchmarks, while maintaining or even improving retrieval accuracy.

Technology Category

• Search and Optimization: Distributed Search
• Machine Learning: Graph-based Machine Learning
• Computer Vision: Image and Video Retrieval

Application Category

• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

In multi-vector retrieval, both queries and data are represented as sets of high-dimensional vectors, enabling finer-grained semantic matching and improving retrieval quality over single-vector approaches. However, its practical adoption is held back by the lack of effective indexing algorithms. Existing work, attempting to reuse standard single-vector indexes, often fails to preserve multi-vector semantics or remains slow. In this work, we present GEM, a native indexing framework for multi-vector representations. The core idea is to construct a proximity graph directly over vector sets, preserving their fine-grained semantics while enabling efficient navigation. First, GEM designs a set-level clustering scheme. It associates each vector set with only its most informative clusters, effectively reducing redundancy without hurting semantic coverage. Then, it builds local proximity graphs within clusters and bridges them into a globally navigable structure. To handle the non-metric nature of multi-vector similarity, GEM decouples the graph construction metric from the final relevance score and injects semantic shortcuts to guide efficient navigation toward relevant regions. At query time, GEM launches beam search from multiple entry points and prunes paths early using cluster cues. To further enhance efficiency, a quantized distance estimation technique is used for both indexing and search. Across in-domain, out-of-domain, and multi-modal benchmarks, GEM achieves up to 16x speedup over state-of-the-art methods while matching or improving accuracy.

Problem

• multi-vector retrieval
• indexing algorithms
• proximity graph
• semantic matching
• high-dimensional vectors

Innovation

• multi-vector retrieval
• proximity graph
• set-level clustering
• non-metric similarity
• quantized distance estimation