PMMC: Prospective Multimodal Memory Compilation for Long-Term LVLM Agents

2026-08-01

AI Summary

This work addresses the inefficiency and poor robustness of existing large vision-language model (LVLM) agents in handling queries requiring image-text binding, temporal updates, or fine-grained visual information due to limitations in their long-term memory systems. To overcome this, the authors propose a forward-looking multimodal memory compilation framework that anticipates potential queries during the memory consolidation phase, compiles conditional memory programs, and validates evidence pathways through a questioning mechanism. Introducing the novel “memory-as-computation” paradigm, the framework employs a synergistic architecture integrating a question generator, planner, and skeptic to construct a structured question bank for efficient retrieval. Experiments demonstrate that this approach significantly improves answer quality and visual evidence recall on multimodal long-term memory benchmarks while reducing token consumption and latency at query time.

Abstract

Long-term memory is essential for LVLM agents to maintain consistency and integrate information across extended multimodal interactions. Existing agent memory systems, however, often reduce visual experiences into textual summaries or rely on static retrieve-then-reason pipelines, which are inefficient at query time and brittle when questions require image-text binding, temporal updates, or visual details. We propose Prospective Multimodal Memory Compilation, a framework that shifts part of the memory reasoning process from query time to memory consolidation time. Given accumulated multimodal interactions, a Questioner predicts future question candidates, a Planner compiles question-conditioned multimodal memory programs, and a Doubter verifies whether the planned evidence path can support the predicted answer. The verified question-program pairs form a structured question bank for efficient query-time routing and evidence retrieval. Experiments on multimodal long-term memory benchmarks show that our method improves answer quality and visual evidence recall while reducing query-time token and latency costs. Extensive ablations analyze the effects of self-feedback, dynamic planning, raw-image access, and question bank coverage.

Problem

• long-term memory
• multimodal interaction
• image-text binding
• query efficiency
• visual detail

Innovation

• Prospective Memory Compilation
• Multimodal Long-Term Memory
• Query-Time Efficiency
• Memory Consolidation
• LVLM Agents