AMES: Approximate Multi-modal Enterprise Search via Late Interaction Retrieval

2026-03-13

AI Summary

This work addresses the challenge of multimodal retrieval across text, images, and videos in enterprise search systems by proposing a unified late-interaction framework that operates without requiring modifications to existing backend architectures. The approach employs a multi-vector encoder to map heterogeneous modalities into a shared representation space and integrates a two-stage retrieval pipeline—comprising approximate nearest neighbor (ANN) candidate generation followed by accelerator-optimized exact MaxSim reranking—to enable efficient, fine-grained cross-modal search. As the first backend-agnostic multimodal solution deployed in a production system, it demonstrates strong empirical effectiveness on the ViDoRe V3 benchmark and achieves competitive ranking performance within a scalable Solr-based infrastructure.

Technology Category

Computer Vision: Multi-modal VisionSearch and Optimization: Distributed SearchMachine Learning: Multimodal Learning

Application Category

Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAGUser Modeling, Personalization and Recommendation: Fairness-aware retrieval and rankingWeb Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

We present AMES (Approximate Multimodal Enterprise Search), a unified multimodal late interaction retrieval architecture which is backend agnostic. AMES demonstrates that fine-grained multimodal late interaction retrieval can be deployed within a production grade enterprise search engine without architectural redesign. Text tokens, image patches, and video frames are embedded into a shared representation space using multi-vector encoders, enabling cross-modal retrieval without modality specific retrieval logic. AMES employs a two-stage pipeline: parallel token level ANN search with per document Top-M MaxSim approximation, followed by accelerator optimized Exact MaxSim re-ranking. Experiments on the ViDoRe V3 benchmark show that AMES achieves competitive ranking performance within a scalable, production ready Solr based system.

Problem

• multimodal retrieval
• enterprise search
• late interaction
• cross-modal search
• production deployment

Innovation

• late interaction retrieval
• multimodal search
• multi-vector encoding
• MaxSim approximation
• enterprise search