Linear Multi-Timescale Retention as a Memory-Efficient Vision-Language Bridge

2026-08-02

AI Summary

This work addresses the O(N²) memory bottleneck of Softmax-based attention in high-resolution vision-language models, which hinders global scene understanding and object constancy. Existing linear attention methods fail to preserve spatial sequence routing, leading to degraded performance. To overcome this, we propose the Linear Multi-timescale Memory Retention (LIA-MTR) module, which leverages ELU-based positive feature mappings, adaptive write-in gating, and a log-linear recurrent decay mechanism to enable lossless long-sequence routing and cross-modal fusion under strict O(N) complexity. LIA-MTR supports unbounded context extension and substantially mitigates the “lost-in-the-middle” phenomenon. Experiments demonstrate that LIA-MTR processes 262,144 visual tokens using only 11.2 GB of GPU memory—far exceeding the 16,384-token limit of standard multi-head attention—and achieves a 71.00% MME score, outperforming an MLP baseline (68.11%) with a 10-percentage-point gain in object constancy.

Abstract

Vision-Language Models (VLMs) face a critical computational bottleneck when processing high-resolution imagery due to the $O(N^2)$ memory complexity of Softmax Multi-Head Attention (MHA). While substituting MHA with independent Multi-Layer Perceptrons (MLPs) achieves $O(N)$ scaling, it strips the architecture of spatial sequence routing, severely degrading global scene understanding and object permanence. In this paper, we propose the Linear Multi-Timescale Retention (LIA-MTR) module, a memory-efficient cross-modal bridge. By integrating an ELU-based positive feature mapping with adaptive write-gating and log-linearly distributed recurrent decays, LIA-MTR mathematically compresses continuous visual sequences into bounded memory states. Theoretical analysis proves the architecture operates with strict $O(N)$ sequence-interaction complexity. Empirically, synthetic retrieval evaluations demonstrate that LIA-MTR flawlessly routes context across 16,000 tokens, eliminating the "Lost in the Middle" degradation typical of naive linear attention. Hardware benchmarking reveals infinite-context scaling capabilities, natively processing 262,144 visual patches within an 11.2 GB VRAM footprint, whereas standard MHA suffers out-of-memory failure at 16,384 patches. Furthermore, following instruction tuning on 665K conversational samples, LIA-MTR significantly outperforms an industry-standard MLP baseline on the MME benchmark (71.00% vs. 68.11%), driven by a 10% absolute improvement in object permanence and superior global semantic extraction. This work establishes a mathematically rigorous, computationally flat foundation for infinite-context Vision-Language integration.

Problem

• Vision-Language Models
• Memory Efficiency
• Multi-Head Attention
• Sequence Modeling
• Computational Complexity

Innovation

• Linear Multi-Timescale Retention
• Memory-Efficient Attention
• Vision-Language Models
• O(N) Complexity
• Infinite Context