Continuous Memory Machines

2026-10-06

AI Summary

This study addresses the limitation of traditional recurrent neural networks, which compress short-term computation and long-term memory into a single state, thereby lacking the synergistic capacity for rapid processing and persistent retention observed in biological neurons. To overcome this, we propose the Continuous Memory Machine (CMM), which constructs matrix-valued short- and long-term memory states based on continuous thinking mechanisms and leverages Transformers to enable bidirectional read-write operations and independent updates. This work represents the first integration of neuron-level parameterized models with persistent long-term memory, effectively balancing rapid computation with information storage. Empirical evaluations demonstrate that CMM surpasses baseline methods on algorithmic and reasoning tasks, exhibits superior generalization compared to existing augmented networks, and preserves interpretable attention patterns.

Abstract

Recurrent neural networks typically compress information into a single vector-valued recurrent state, forcing short-term computation and long-term retention to share the same representation. Past extensions alleviate this bottleneck by increasing the memory capacity or separating timescales, but lack the combination of rapid neuron-level processing and longer-term retention found in biology. To that end, we introduce the Continuous Memory Machine (CMM), a recurrent architecture with matrix-valued short- and long-term memory states serving distinct functional roles. Building on the Continuous Thought Machine (CTM), the CMM's short-term memory tracks recent neural activity, with uniquely parameterized neuron-level models learning to use these activity patterns for computation. A persistent long-term memory stores information for later use, with a Transformer jointly updating both memory stores, providing an expressive bidirectional read--write mechanism such that each store can reorganize its own contents and both read from and write to the other. Across algorithmic, in-context learning, and recurrent reasoning tasks, the CMM outperforms a broad suite of baselines, exhibiting stronger generalization than prior memory-augmented networks while preserving the CTM's interpretable attention patterns. Code is available at https://github.com/SakanaAI/continuous-memory-machines.

Problem

• Recurrent Neural Networks
• Memory Representation
• Short-term Computation
• Long-term Retention
• Memory-augmented Networks

Innovation

• Continuous Memory Machine
• matrix-valued memory states
• neuron-level processing
• bidirectional read-write mechanism
• memory-augmented networks