Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning

2026-10-08

AI Summary

This study addresses the prohibitive computational overhead of processing long visual sequences in multimodal large language models and the inability of fixed compression to accommodate dynamic reasoning. To this end, it proposes ViMoD, a framework that enables efficient on-demand inference by maintaining a compact visual context while preserving access to original fine-grained evidence. Methodologically, ViMoD innovatively integrates content-adaptive grouped aggregation with decoding-history-based temporal routing. Specifically, it employs Deformable Area Token Aggregation (DART) to construct coarse-grained representations and Temporal Routing for Adaptive Context Evidence (TRACE) to dynamically select, retain, or replace active token groups. Experimental results demonstrate that, when instantiated on Qwen3-VL-4B, ViMoD surpasses all baselines using only a 20% token budget, achieving a 39.0% improvement in average normalized score while introducing fewer than 0.06% additional parameters.

Abstract

Processing long visual token sequences from high-resolution images makes multi-step reasoning computationally expensive for multimodal Large Language Models (MLLMs). Existing one-shot pruning and aggregation methods compress visual tokens into a fixed context before decoding. However, visual evidence needs can shift as reasoning unfolds, making it difficult for a fixed compressed context to retain all the details needed across stages. To address this challenge, we propose ViMoD, a lightweight framework that maintains a compact visual context while preserving access to original fine-grained evidence as reasoning needs evolve. Deformable Aggregation of Region-wise Tokens (DART) learns content-adaptive groups and aggregation capacities, constructing compact Coarse representations linked to recoverable original Fine tokens. Temporal Routing for Adaptive Contextual Evidence (TRACE) integrates decoding history to anticipate upcoming evidence needs and select, retain, or replace active Fine-token groups. Selected Fine tokens augment the persistent Coarse context in the frozen backbone, enabling stage-specific evidence access without continuously attending to all visual tokens. On Qwen3-VL-4B, ViMoD outperforms all evaluated baselines on all eight reasoning benchmarks at a 20% target visual token budget, improving the mean normalized score by 39.0% over the strongest evaluated one-shot baseline. These gains are achieved with only 0.0546% additional trainable parameters relative to the frozen backbone.

Problem

• Multimodal Large Language Models
• Visual Token Compression
• Multi-step Reasoning
• Computational Efficiency
• Dynamic Visual Evidence

Innovation

• Multimodal Reasoning
• Visual Token Compression
• Deformable Aggregation
• Temporal Routing
• On-demand Visual Memory