Learning Successor Features with Distributed Hebbian Temporal Memory

2023-10-20

International Conference on Learning Representations

AI Summary

To address online sequential learning and uncertainty-aware decision-making under nonstationary, partially observable environments, this paper proposes the Distributed Hebbian Temporal Memory (DHTM) model. DHTM is built upon a factor graph formalism and a multi-compartmental neuron architecture, integrating distributed representations, sparse state transitions, and local Hebbian plasticity rules to enable online, stable learning and cumulative prediction of successor features (SFs). Its key contribution lies in the first systematic incorporation of biologically plausible Hebbian learning into the SFs framework—preserving interpretability while overcoming critical limitations of RNN- and HMM-based models, such as catastrophic forgetting and slow convergence in dynamic settings. Empirical evaluation on multiple nonstationary benchmark datasets demonstrates that DHTM significantly outperforms LSTM, RWKV, and the biologically inspired HMM variant CSCG, exhibiting superior online adaptability, predictive accuracy, and long-horizon planning capability.

Abstract

This paper presents a novel approach to address the challenge of online sequence learning for decision making under uncertainty in non-stationary, partially observable environments. The proposed algorithm, Distributed Hebbian Temporal Memory (DHTM), is based on the factor graph formalism and a multi-component neuron model. DHTM aims to capture sequential data relationships and make cumulative predictions about future observations, forming Successor Features (SFs). Inspired by neurophysiological models of the neocortex, the algorithm uses distributed representations, sparse transition matrices, and local Hebbian-like learning rules to overcome the instability and slow learning of traditional temporal memory algorithms such as RNN and HMM. Experimental results show that DHTM outperforms LSTM, RWKV and a biologically inspired HMM-like algorithm, CSCG, on non-stationary data sets. Our results suggest that DHTM is a promising approach to address the challenges of online sequence learning and planning in dynamic environments.

Problem

• Online sequence learning in non-stationary environments
• Decision making under partial observability
• Overcoming instability in temporal memory algorithms

Innovation

• Distributed Hebbian Temporal Memory for sequence learning
• Factor graph and multi-component neuron model
• Sparse matrices and local Hebbian learning rules