temporal memory retrieval

Modeling long-range temporal dependencies in high-dimensional sequential data (e.g., video frames) remains challenging for memory-intensive tasks. Method: This paper proposes a memory architecture integrating temporal kernels with dense Hopfield networks. It explicitly encodes temporal bias via a learnable temporal kernel function $K(m,k)$ and establishes a higher-order interaction mechanism grounded in energy-functional optimization, theoretically enabling exponential memory capacity and supporting efficient, continuous retrieval over long sequences. Crucially, it augments the self-attention mechanism—unlike standard Transformers—with a trainable, temporally aware memory module. Contribution/Results: Experiments on movie frame storage and ordered reconstruction demonstrate significant performance gains over baselines, effectively alleviating the long-range dependency bottleneck. The approach offers a novel paradigm for video understanding and long-context modeling in sequence learning.

Traditional recurrent models struggle to effectively retain early contextual information in long sequences due to their fixed-size hidden states. This work proposes a state anchoring mechanism that periodically caches recurrent states as an expandable memory and generates content-conditional anchor keys for each cached state, enabling efficient retrieval through causal attention. The approach significantly enhances long-range memory capacity while maintaining computational efficiency. Experimental results demonstrate that the proposed method outperforms various linear attention variants across commonsense reasoning, LongBench, and context retrieval tasks, effectively improving the long-context modeling capabilities of recurrent architectures.

To address context memory decay, factual inconsistency, and reduced coherence in large language models (LLMs) when processing long sequences, this paper proposes the Hierarchical Latent State Reweaving (HLSR) mechanism. HLSR systematically strengthens cross-layer long-range dependency modeling—without introducing additional parameters, external memory, or modifications to the attention architecture—by hierarchically reconstructing latent states and reweighting contextual representations in the hidden space. Its core innovations are a lightweight state fusion operator and structured analysis of attention distributions to guide latent state reweaving. Experiments demonstrate significant improvements: +12.7% recall accuracy on long-text generation and ambiguous query tasks, +23.4% retention rate for rare tokens, and −18.1% error rate in numerical reasoning, all with <5% increase in computational overhead.

Long-sequence modeling faces fundamental challenges including limited context length, difficulty in capturing long-range dependencies, and low efficiency in online learning. To address these, this work systematically reviews the resurgence of state-space models (SSMs) and recurrent computation, proposing a novel local forward-computation paradigm tailored for real-world online learning—thereby circumventing the temporal backtracking constraints inherent in standard backpropagation through time (BPTT). We introduce the first unified taxonomy encompassing both deep SSMs and large-context Transformers. Our framework integrates structured linear attention, enhanced RNN architectures, local recurrence mechanisms, and online optimization algorithms. The study rigorously clarifies the theoretical representational advantages and practical sequential reasoning benefits of recurrent modeling over alternatives. Collectively, this work delivers a scalable technical roadmap for low-latency, highly extensible long-sequence modeling.

We identify and prove a fundamental trade-off governing long-sequence models: no model can simultaneously achieve (i) per-step computation independent of sequence length (Efficiency), (ii) state size independent of sequence length (Compactness), and (iii) the ability to recall a number of historical facts proportional to sequence length (Recall). We formalize this trade-off within an Online Sequence Processor abstraction that unifies Transformers, state space models, linear recurrent networks, and their hybrids. Using the Data Processing Inequality and Fano's Inequality, we prove that any model satisfying Efficiency and Compactness can recall at most O(poly(d)/log V) key-value pairs from a sequence of arbitrary length, where d is the model dimension and V is the vocabulary size. We classify 52 architectures published before March 2026 into the triangle, showing that each achieves at most two of the three properties and that hybrid architectures trace continuous trajectories in the interior. Experiments on synthetic associative recall tasks with five representative architectures validate the theoretical bound: empirical recall capacity lies strictly below the information-theoretic limit, and no architecture escapes the triangle.

This work addresses the inefficiency of traditional token-level context modeling in distinguishing between recollective, summarizing, and local information. The authors propose a novelty-driven memory mechanism that dynamically partitions context into three components: a content-addressable novelty cache for retrievable details, a recurrent state for compressed summaries, and a sliding window for recent local context. This architecture uniquely scales memory capacity with the amount of distinct information rather than raw token count, yielding an auditable and interpretable working memory structure. Integrating a Dirichlet-process-inspired novelty-gated attention mechanism, the system achieves full-attention performance in character-level control tasks with roughly half the attention cost and outperforms both full-attention and fixed-budget baselines on a thousand-event healthcare claims prediction task, while enabling human inspection of stored memory contents.