Coverage Matters: MarginMerge for Compressing Multi-Vector Visual Document Retrievers

2026-08-03

AI Summary

This work addresses the challenge of large index size and high post-retrieval computational cost in multi-vector visual document retrieval, where existing compression methods often neglect query-aware coverage of relevant regions. The authors propose MarginMerge, a training-free index-time compression method that prioritizes preserving diverse document regions likely to serve as top matches across varying queries, rather than selecting patches solely based on saliency. By integrating coverage-aware anchor selection, patch clustering, and a lightweight shared network to synthesize cluster representative vectors, MarginMerge significantly compresses the index while maintaining the standard MaxSim retrieval interface. At vector retention rates of 5%–10%, it achieves state-of-the-art query-agnostic average performance—retaining 97%–99% of nDCG@5, reducing index size by 90%–95%, and decreasing ranking inversions by 41% on average compared to geometric merging at 5% retention.

Abstract

Multi-vector visual document retrievers such as ColPali and ColQwen achieve strong retrieval by storing fine-grained patch embeddings, but this produces large indexes and costly late-interaction scoring. We argue that effective compression should preserve query-relevant coverage, meaning the diverse document regions that may become the strongest MaxSim match across queries, rather than selecting patches independently by salience. This view also explains why dense rendered pages are easier to compress than natural images. We introduce MarginMerge, a compression method for frozen multi-vector retrievers. It selects coverage-aware anchors, clusters document patches, and uses a lightweight shared network to synthesize one representative per cluster. Compression is performed once during indexing, while retrieval keeps the standard MaxSim interface. Across six datasets on both ColQwen2.5 and ColPali, MarginMerge achieves the highest matched query-agnostic average at 5% and 10% vector retention. Compared with the uncompressed index using the same backbone, it preserves between 97% and 99% of average nDCG@5 while reducing stored document vectors by between 90% and 95%. At 5% retention, it also reduces ranking flips relative to geometric merging on all six ColQwen2.5 datasets by approximately 41% on average. The same model transfers to unseen datasets and retention ratios without retraining.

Problem

• multi-vector retrieval
• visual document retrieval
• index compression
• coverage preservation
• late-interaction scoring

Innovation

• coverage-aware compression
• multi-vector retriever
• MarginMerge
• MaxSim
• vector quantization