Voltic: Distinguishing Volatility from Stochasticity in Recurrent Memory

2026-10-05

AI Summary

This study addresses the suboptimal write policies in recurrent memory models arising from their inability to distinguish association change rates from observation noise. To overcome this, we propose Voltic, a Bayesian filtering-based mechanism that models uncertainty via anisotropic covariances to optimize memory updates. We further design diagonal and quasi-diagonal assumed density approximations that preserve the advantages of vector-valued writes while remaining compatible with parallel training. Computational efficiency is enhanced by integrating the gated Delta rule with chunked kernel acceleration algorithms. Experimental results demonstrate that our approach significantly outperforms baselines on controlled recall tasks, effectively improving both reasoning performance in language models and retrieval accuracy over long contexts.

Abstract

Recurrent sequence models must decide how strongly to overwrite their memory at each token. Read as Bayesian filtering, this write is the gain of a Kalman update, set by uncertainty from two sources that pull it in opposite directions: volatility, how quickly the underlying associations change, and stochasticity, how noisy each observation of them is. First, we show that the update of gated delta-rule memories is the form this filter takes under isotropic uncertainty. Next, we introduce Voltic, a recurrent memory that keeps the covariance anisotropic and makes both noise variances input-dependent, so the write is vector-valued and carries uncertainty accumulated over the sequence. A dense covariance would have to be propagated token by token, ruling out the parallel training these models depend on. We therefore give two assumed-density approximations, diagonal and quasi-diagonal, both of which leave the memory update in delta-rule form and reuse its chunked kernels. On controlled recall tasks in which associations change and observations are corrupted, Voltic leads all baselines. On the task combining volatility and stochasticity, its margin over the strongest baseline is larger at both extrapolation sizes than at the training sizes. In 45M-parameter language models it leads an eight-task reasoning average and achieves higher retrieval accuracy beyond the training context length than gated baselines, at throughput close to those baselines. Deriving the write from an uncertainty recursion therefore makes memory more responsive to change.

Problem

• recurrent memory
• volatility
• stochasticity
• Bayesian filtering
• uncertainty

Innovation

• recurrent memory
• Bayesian filtering
• Kalman update
• volatility-stochasticity disentanglement
• assumed-density approximation