Beyond Retrieval: Analytic Memory for Multimodal Agents

2026-07-31

AI Summary

Existing long-term multimodal memory systems primarily focus on retrieval and lack the capacity for computational analysis across interactions. This work proposes an analytical memory mechanism that introduces, for the first time, a schema-free attribute-value abstraction of observations, automatically extracting and materializing structured memory from conversations, images, and metadata. The mechanism supports analytical operations such as filtering, aggregation, sorting, and temporal comparison, and unifies retrieval with analysis through the AdaMM framework. Evaluated on the MemEye and MemGallery benchmarks, the proposed approach achieves performance gains of 11.3% and 7.3%, respectively, substantially enhancing the flexibility and analytical capabilities of multimodal memory systems.

Abstract

Long-term multimodal memory must support not only retrieving relevant information but also computing over observations accumulated across interactions. Existing systems largely emphasize \emph{retrieval memory}, organizing interaction histories through summaries and indexes to return query-relevant information at multiple granularities, from high-level abstractions to underlying records. In this paper, we formulate \emph{analytic memory} as a complementary abstraction that organizes recurring multimodal observations into queryable structures supporting filtering, aggregation, ranking, and temporal comparison. We present AdaMM, a framework that jointly supports retrieval and analytic memory. Rather than relying on application-defined schemas, AdaMM extracts provenance-linked attribute-value observations from dialogue, images, and contextual metadata, discovers recurring field structures, and materializes them for analytical access. At inference time, a memory-aware planner decomposes queries into retrieval and analytic operations and routes each operation to the appropriate tools. Experiments on two long-term multimodal memory benchmarks, MemEye and MemGallery, show that AdaMM improves performance by up to 11.3\% and 7.3\%, respectively.

Problem

• multimodal memory
• analytic memory
• retrieval memory
• long-term memory
• memory abstraction

Innovation

• analytic memory
• multimodal agents
• schema-free structuring
• memory-aware planning
• provenance-linked observations