PilotANN: Memory-Bounded GPU Acceleration for Vector Search

2025-03-27

AI Summary

To address the dual challenges of GPU memory constraints and CPU computational bottlenecks in large-scale, high-dimensional Approximate Nearest Neighbor Search (ANNS), this paper proposes a CPU-GPU collaborative hybrid architecture. Our method decomposes graph traversal into three stages, introduces a fast dynamic entry-point selection strategy to optimize initialization, and—novelly—enables seamless coordination between SVD-compressed vector traversal on GPU and full-precision vector refinement on CPU, augmented by memory-aware subgraph scheduling. Evaluated on billion-scale datasets, our approach achieves 3.9–5.4× higher throughput over baselines and supports ANNS on datasets 12× larger than GPU memory capacity. All code is publicly available.

Technology Category

• Machine Learning: Hardware-aware ML
• Search and Optimization: Distributed Search
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Approximate Nearest Neighbor Search (ANNS) has become fundamental to modern deep learning applications, having gained particular prominence through its integration into recent generative models that work with increasingly complex datasets and higher vector dimensions. Existing CPU-only solutions, even the most efficient graph-based ones, struggle to meet these growing computational demands, while GPU-only solutions face memory constraints. As a solution, we propose PilotANN, a hybrid CPU-GPU system for graph-based ANNS that utilizes both CPU's abundant RAM and GPU's parallel processing capabilities. Our approach decomposes the graph traversal process of top-$k$ search into three stages: GPU-accelerated subgraph traversal using SVD-reduced vectors, CPU refinement and precise search using complete vectors. Furthermore, we introduce fast entry selection to improve search starting points while maximizing GPU utilization. Experimental results demonstrate that PilotANN achieves $3.9 - 5.4 imes$ speedup in throughput on 100-million scale datasets, and is able to handle datasets up to $12 imes$ larger than the GPU memory. We offer a complete open-source implementation at https://github.com/ytgui/PilotANN.

Problem

• Hybrid CPU-GPU system for memory-bounded vector search
• Accelerates graph-based ANNS with SVD and parallel processing
• Handles large datasets exceeding GPU memory limits

Innovation

• Hybrid CPU-GPU system for graph-based ANNS
• SVD-reduced vectors for GPU subgraph traversal
• Fast entry selection to maximize GPU utilization