Naju: A Native Discrete State-Space Model with Independent Retention and Writing for Long-Sequence Memory

2026-07-23

AI Summary

Existing long-sequence memory models struggle to simultaneously achieve lossless long-term retention and effective overwriting of outdated information. This work proposes Naju, the first model to decouple forgetting and writing mechanisms within a discrete state space. By employing a learnable sigmoid forget gate, an independent write gate, and input-dependent linear read-write mappings, Naju decomposes recurrent updates into orthogonal operations. This design overcomes the theoretical trade-off between retention rate and write strength inherent in conventional single-gate architectures, all while preserving linear time and space complexity. Experiments demonstrate that Naju maintains superior memory retention and overwrite capabilities even when trained on sequences four times longer than baseline lengths, outperforming Mamba on WikiText-103, the Long Range Arena benchmark, and multi-query associative recall tasks, with performance comparable to Transformers.

Abstract

Long-sequence memory tracking places two opposing demands on a recurrent state: near-lossless retention of stored bindings over long horizons, and active overwriting of stale ones. In our diagnostic suite, the strongest efficient baselines tend to solve only one side well. Continuous-time-parameterized state-space models (SSMs) such as Mamba obtain their discrete recurrence by zero-order-hold discretization of a continuous-time system; we argue that this detour is unnecessary for memory tracking and parameterize the discrete transition directly. Naju (Native Adaptive Junction Unit) factorizes the recurrent update, schematically $x_n = f_n\odot x_{n-1} + i_n\odot(B_n u_n)$, into an explicit discrete pole (a learned forget gate $f_n$), an independent write gain $i_n$, and input-dependent write/read maps. Since the sigmoid pole satisfies $0<f_n<1$, each frozen local coordinate is Schur-stable by construction, and the full time-varying recurrence satisfies a fading-memory/BIBO bound under uniform boundedness assumptions, with no stability regularizer. We formalize the key structural limitation of coupled designs: any non-expansive complementary single-gate recurrence ties the effective retention $r$ and write gain $w$ through $|r|+w\le 1$, so near-complete retention forces weak writing; decoupling $f_n$ from $i_n$ removes this constraint. Empirically, Naju is the only evaluated model that remains strong on both retention and overwriting at 4x the training length. Beyond the diagnostic suite, we evaluate Naju on WikiText-103 language modeling, Long Range Arena, and multi-query associative recall. Across these settings, Naju consistently combines strong long-range memory with competitive or superior performance, outperforming the Mamba baselines in the principal comparisons while remaining competitive with the Transformer and preserving linear-time, linear-memory scaling.

Problem

• long-sequence memory
• memory retention
• memory overwriting
• recurrent state
• state-space models

Innovation

• state-space model
• memory retention
• decoupled gating
• discrete recurrence
• long-sequence modeling