ReMAP: Restoring the Perceptual Cycle with Reasoning-Time Latent Visual Memory

2026-10-04

AI Summary

This study addresses the degradation of visual grounding in multimodal large language models during extended reasoning, caused by visual attention decay. To mitigate this, we propose a dual complementary latent visual memory mechanism triggered by reasoning states. This approach couples static global memory with dynamic local memory and incorporates a reinforcement learning-based access policy to retrieve compact latent visual evidence on demand throughout the reasoning process. By reconstructing the perception loop, it achieves synergy between global context anchoring and dynamic re-encoding of local evidence. Experimental results demonstrate that our method significantly outperforms existing approaches across multiple multi-image benchmarks, achieving improvements of up to 14.84%, while reducing visual token consumption by 51% to 76.8%.

Abstract

As multimodal large language models (MLLMs) reason for longer, attention to the initial visual input diminishes, weakening visual grounding. Visual memory reintroduces visual evidence during reasoning. We conduct a controlled analysis of visual memory along three axes: curation, organization, and access. We find that local evidence benefits from global context, compact latent representations balance accuracy and visual-context cost, and the utility of memory access depends on the reasoning state. Guided by these findings, we propose ReMAP (Reasoning-Time Memory-Augmented Perception), which couples two complementary latent memories: a static, question-conditioned Global memory that preserves scene and cross-image context, and a dynamic Local memory that uses this context as an anchor while selecting and re-encoding region-level evidence according to the current reasoning state. Both memories return compact latent tokens inserted into the reasoning sequence, and a reinforcement-learning access policy trained with branched rollouts decides when to continue reasoning or invoke Global or Local memory. On ten benchmark families, ReMAP outperforms prior visual-memory methods on all four multi-image benchmarks, exceeding the strongest prior results on MuirBench and MIMIC by 8.38 and 14.84 percentage points. Across four backbone families, enabling memory access improves over the same trained model with memory disabled, and on shared V*Bench, CV-Bench-2D, and MuirBench questions ReMAP reduces the visual tokens entering the reasoning sequence by 51.0-76.8% relative to the native-resolution backbone. Further analyses show that Global and Local memory form distinct yet complementary latent representations. Together, these components restore the perceptual cycle by letting the reasoning state trigger targeted visual retrieval, with the retrieved evidence guiding subsequent reasoning.

Problem

• Multimodal Large Language Models
• Visual Grounding
• Visual Memory
• Long-horizon Reasoning

Innovation

• Multimodal Large Language Models
• Latent Visual Memory
• Reasoning-Time Retrieval
• Reinforcement Learning Access Policy
• Perceptual Cycle