Cracking Vector Search Indexes

2025-03-03

AI Summary

Prebuilding vector indexes over massive cold data in data lakes incurs prohibitive computational cost and latency, severely degrading ANNS efficiency in RAG systems. To address this, we propose CrackIVF—an adaptive, partitioned, hierarchical indexing framework featuring a novel query-driven dynamic index fission mechanism: it eliminates the need for full pre-indexing and instead incrementally optimizes partition topology and cluster centroids in real time using query feedback. CrackIVF integrates three key techniques: incremental IVF clustering, approximate brute-force warm-start initialization, and query-pattern-aware partition merging. Experiments demonstrate that CrackIVF reduces index initialization time by 10×–1000× compared to conventional methods; crucially, it begins serving million-scale queries *before* traditional approaches even complete offline indexing. Moreover, its final index quality matches that of the optimal offline index.

Technology Category

• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Intelligent Query Processing
• Computer Vision: Image and Video Retrieval

Application Category

• Search and Retrieval-Augmented AI: Web crawling and indexing
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Retrieval Augmented Generation (RAG) uses vector databases to expand the expertise of an LLM model without having to retrain it. This idea can be applied over data lakes, leading to the notion of embeddings data lakes, i.e., a pool of vector databases ready to be used by RAGs. The key component in these systems is the indexes enabling Approximated Nearest Neighbor Search (ANNS). However, in data lakes, one cannot realistically expect to build indexes for every possible dataset. In this paper, we propose an adaptive, partition-based index, CrackIVF, that performs much better than up-front index building. CrackIVF starts answering queries by near brute force search and only expands as it sees enough queries. It does so by progressively adapting the index to the query workload. That way, queries can be answered right away without having to build a full index first. After seeing enough queries, CrackIVF will produce an index comparable to the best of those built using conventional techniques. As the experimental evaluation shows, CrackIVF can often answer more than 1 million queries before other approaches have even built the index and can start answering queries immediately, achieving 10-1000x faster initialization times. This makes it ideal when working with cold data or infrequently used data or as a way to bootstrap access to unseen datasets.

Problem

• Improves query response without full index building
• Adapts index to query workload progressively
• Enables immediate querying on cold or infrequent data

Innovation

• Adaptive partition-based index CrackIVF
• Progressive index adaptation to query workload
• Immediate query answering without full index