M$^3$Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions

2026-06-05

AI Summary

Existing benchmarks struggle to evaluate agents’ capabilities in memory retention, cross-modal reasoning, and comprehension of implicit information within authentic multimodal dialogues. To address this gap, this work introduces M³Exam—the first benchmark specifically designed for assessing memory in real-world user-agent multimodal interactions—and proposes M³Proctor, a query-driven, on-demand visual retrieval method. By preserving the semantic integrity of original visual content while substantially improving computational efficiency, M³Proctor effectively mitigates query modality bias, reduces both index construction time and retrieval token count by over 70%, and boosts accuracy by 13%. This approach establishes a new paradigm for memory modeling and evaluation in multimodal large language models.

Abstract

Language agents are increasingly deployed over accumulating multimodal information, yet existing benchmarks assume a human-human form with sparse visuals and straightforward content, evaluating neither reasoning over authentic multimodal file interaction nor the interpretation of concealed user information. We therefore introduce M$^3$Exam, a query-centric multimodal conversational memory benchmark built on realistic user-agent interaction, with multi-dimensional evaluation spanning cross-modal grounding and implicit information inference. Benchmarking MLLMs and memory systems reveals persistent gaps in cross-modal grounding, cross session reasoning, and the efficiency cost of accumulating multimodal context. We further propose M$^3$Proctor, a multimodal memory method that detects query modality bias and consumes raw visual sources only on demand, improving accuracy by 13% while cutting index-construction time and retrieved tokens by over 70%.

Problem

• multimodal memory
• user-agent interaction
• cross-modal grounding
• implicit information inference
• memory benchmark

Innovation

• multimodal memory
• cross-modal grounding
• implicit information inference
• query-centric benchmark
• on-demand visual retrieval