Raven: High-Recall Sequence Modeling with Sparse Memory Routing

2026-07-28

AI Summary

This work addresses the challenge faced by existing linear-time sequence models in long-context recall: state-space models suffer from interference due to dense memory writes, while sliding-window attention mechanisms discard critical information through rigid, fixed-size windows. To overcome this limitation, we propose Raven, a linear-time sequence model that employs an input-dependent sparse memory routing mechanism, updating only a subset of fixed memory slots at each step. By integrating exponential decay with selective update strategies, Raven effectively mitigates both interference and hard information loss, substantially enhancing long-range content retention. Raven is compatible with both linear Transformers and state-space model architectures, demonstrating strong performance on multiple recall-intensive benchmarks and maintaining robust recall capabilities even when extrapolating to contexts 16 times longer than the training sequence length.

Abstract

Long-context recall in linear-time sequence models highlights a tradeoff in how they write to memory. State-based linear models, such as state-space models (SSMs) and linear Transformers, write densely, updating the entire state for each newly arrived token, which leads to interference and makes specific past tokens hard to recover. Sliding-window attention (SWA) exhibits the opposite behavior: it writes sparsely by storing explicit token representations, but only within a fixed window, so recall drops once the relevant token is evicted. Interpolating between these models, we introduce Raven, a linear-time sequence model that maintains a fixed set of memory slots and, at each step, decays and updates only a selected subset via learned, input-dependent routing. This lets Raven mitigate SWA's position-based overwriting and hard eviction while reducing interference from dense state updates in SSMs, thereby preserving long-range content much more effectively. Across recall-intensive benchmarks, Raven is competitive with or outperforms prior linear-time baselines, achieving strong long-context recall where both SWA and SSMs sharply degrade. It remains effective when extrapolating to context lengths as large as 16x its training length, with similar gains in hybrid architectures.

Problem

• long-context recall
• memory interference
• sparse memory routing
• sliding-window attention
• state-space models

Innovation

• sparse memory routing
• long-context recall
• linear-time sequence modeling
• memory slots
• input-dependent routing