Hierarchical BM25: Lexical Search at Billion-Document Scale

2026-07-31

AI Summary

This work addresses the high memory overhead and query latency of traditional BM25 indexing at billion-document scale, which hinders real-time retrieval. The authors propose a hierarchical retrieval paradigm that first employs a lightweight coarse-grained index—constructed via topic-balanced clustering and a term-frequency–co-occurrence dual-signal strategy—to identify relevant document groups, followed by exact BM25 scoring only within the selected candidates. By confining approximation solely to the group selection stage, the approach preserves scoring accuracy while achieving constant memory usage independent of corpus size and millisecond-level response times. On a billion-document corpus, it attains 300 ms latency and 32 queries per second (after cache warm-up), yielding a 4.7–5.6× throughput improvement; on a 500K-document subset, retrieving just 5–10% of clusters recovers 83–92% of the original score recall.

Abstract

A flat BM25 index over one billion documents occupies about 400 GB. Holding it in memory requires DRAM proportional to corpus size. Serving it from disk takes 4-12 seconds per query. Exact top-k lexical retrieval at this scale is therefore impractical within an interactive latency budget. Hierarchical BM25 gives up exact ranking in exchange for fixed bounds on memory and latency. A resident coarse index selects which of ~1K topical, size-balanced document groups a query visits, using two signals: the total frequency of each query term within a group, and, for informative terms spread too thinly across groups for frequency totals to reflect, whether several of them appear together in one document. Selected groups are then searched exhaustively and scored against ~100 KB of global statistics. Every returned score therefore equals the flat index's score, and the approximation is confined to selection alone. The resident footprint is ~4.4 GB, independent of corpus size. Sixteen-term queries over one billion documents return in ~300 ms (4.7x to 5.6x the throughput of a flat multi-threaded index), and a warmed cache sustains ~32 queries per second versus under 3 for flat indexing. At a 500K-document configuration, visiting 5-10% of clusters recovers 0.83-0.92 of the exhaustive result score. Billion-scale recall and a direct comparison against document-reordered BlockMax-WAND remain open.

Problem

• BM25
• lexical search
• billion-scale retrieval
• interactive latency
• top-k retrieval

Innovation

• Hierarchical BM25
• lexical search
• billion-scale retrieval
• approximate top-k
• memory-efficient indexing