On the Spatiotemporal Dynamics of Generalization in Neural Networks

2026-02-02

AI Summary

This work addresses the well-known difficulty of neural networks in achieving length extrapolation on tasks such as addition, a capability humans handle effortlessly across arbitrary sequence lengths. Drawing inspiration from physical principles, the authors identify three necessary conditions for successful generalization: locality, symmetry, and stability. Building upon these, they derive SEAD—an iterative neural cellular automaton governed by local convolutional rules that converge to a fixed point. Notably, SEAD embeds logical reasoning directly into its dynamical architecture rather than relying on scaling model parameters. Experiments demonstrate perfect length extrapolation and divergence-free training on parity, addition (generalizing from length L=16 to L=10⁶ with 100% accuracy), and Rule 110 tasks.

Technology Category

• Search and Optimization: Learning to Search
• Multiagent Systems: Adversarial Agents
• Machine Learning: Deep Neural Architectures and Foundation Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Why do neural networks fail to generalize addition from 16-digit to 32-digit numbers, while a child who learns the rule can apply it to arbitrarily long sequences? We argue that this failure is not an engineering problem but a violation of physical postulates. Drawing inspiration from physics, we identify three constraints that any generalizing system must satisfy: (1) Locality -- information propagates at finite speed; (2) Symmetry -- the laws of computation are invariant across space and time; (3) Stability -- the system converges to discrete attractors that resist noise accumulation. From these postulates, we derive -- rather than design -- the Spatiotemporal Evolution with Attractor Dynamics (SEAD) architecture: a neural cellular automaton where local convolutional rules are iterated until convergence. Experiments on three tasks validate our theory: (1) Parity -- demonstrating perfect length generalization via light-cone propagation; (2) Addition -- achieving scale-invariant inference from L=16 to L=1 million with 100% accuracy, exhibiting input-adaptive computation; (3) Rule 110 -- learning a Turing-complete cellular automaton without trajectory divergence. Our results suggest that the gap between statistical learning and logical reasoning can be bridged -- not by scaling parameters, but by respecting the physics of computation.

Problem

• generalization
• neural networks
• spatiotemporal dynamics
• systematicity
• length generalization

Innovation

• length generalization
• neural cellular automaton
• attractor dynamics
• spatiotemporal inductive bias
• physics of computation