M$^3$Eval: Multi-Modal Memory Evaluation through Cognitively-Grounded Video Tasks

2026-06-03

AI Summary

Current video understanding benchmarks primarily emphasize perception and reasoning, yet lack systematic evaluation of multimodal models’ memory capabilities. This work introduces the first cognitive psychology–inspired framework for assessing multimodal memory, employing cognitively grounded task designs to disentangle and evaluate a model’s ability to retain information, resist interference, and maintain symbolic representations across spatiotemporal domains in long-form videos. Experimental results reveal that prevailing models struggle to sustain disentangled representations across concurrent video streams, exhibit interference patterns markedly distinct from those observed in humans, and demonstrate limited symbolic memory capacity. These findings provide critical insights for advancing the design of memory mechanisms in multimodal systems.

Abstract

As multi-modal models advance towards long-form video understanding, memory emerges as a critical capability. Despite substantial efforts in developing video datasets and benchmarks, existing works primarily focus on perception and reasoning, without systematically evaluating memory: what models retain, how faithfully information is preserved, and how robust memory remains under interference. To address this gap, we introduce M$^3$Eval, the first comprehensive evaluation framework and benchmark for probing different memory dimensions in multi-modal models. Grounded in cognitive psychology, our design features carefully constructed tasks that isolate key aspects of memory. Leveraging M$^3$Eval, we conduct extensive experiments across representative multi-modal models, revealing consistent weaknesses and distinctive behaviors. We find that models struggle to maintain disentangled representations when processing parallel video streams, exhibit interference patterns differing substantially from those observed in human memory, ground memory sources more reliably in the spatial domain than the temporal domain, and demonstrate limited symbolic memory. Collectively, our benchmark provides a valuable resource for future research, while our findings highlight memory as a fundamental yet underexplored capability and offer insights for designing more effective memory mechanisms in multi-modal models. Our code and dataset are available at https://pku-value-lab.github.io/m3eval-homepage.

Problem

• multi-modal memory
• video understanding
• memory evaluation
• cognitive psychology
• memory robustness

Innovation

• multi-modal memory
• cognitive evaluation
• video understanding
• memory benchmark
• interference robustness