Length-Independent State Tracking Under a Parallel Scan

2026-09-26

AI Summary

This study addresses state error accumulation in parallel scans under finite precision and the limited long-sequence expressivity of linear models by proposing the Neural Finite State Machine (NFSM). We first prove that affine recurrences can only realize deterministic automata, thereby reframing parallel scans as a computational budget rather than a constraint, and design a non-affine dynamical architecture featuring both contractive and separating properties. By integrating non-affine recurrent networks with algebraic structure modeling, our approach achieves length-independent exact state tracking and suppresses numerical perturbations. Experiments demonstrate that a single-layer NFSM learns precise transition tables, while multi-layer stacking maintains perfect cross-length accuracy across Abelian groups, non-invertible semigroups, and text-based tasks.

Abstract

Learning robust and scalable finite-state tracking is fundamental to sequence processing. While linear recurrent neural networks (RNNs), linear attention, and state space models enable scalable parallel training through affine recurrences, their theoretical expressivity guarantees assume idealized arithmetic and do not extend to finite precision, where the parallel scan that makes them fast is itself a source of perturbation. We formalize finite-state tracking at finite precision and characterize length independence: tracking that stays correct at every sequence length, at a precision cost that does not grow with the length. We show that length-independent state tracking requires two competing dynamics within a single map: contraction to suppress numerical perturbations and separation to keep distinct states apart. We prove that affine recurrences, which offer a single rate at each step to serve both roles, realize at most definite automata at finite precision. Instead of treating scan compatibility as a restriction on the update map, we reinterpret it as a computational budget and introduce the Neural Finite-State Machine (NFSM): a nonaffine, scan-compatible RNN built for length-independent finite-state tracking. On synthetic benchmarks spanning abelian and nonabelian groups, noninvertible monoids, and textual state-tracking tasks, affine baselines fail on every nondefinite task, most of them within a few hundred steps. A single NFSM layer instead learns the exact transition tables of every algebraic task, which certifies correctness beyond the tested lengths, and a stack of NFSMs keeps perfect accuracy on the textual tasks at every tested length.

Problem

• finite-state tracking
• parallel scan
• finite precision
• length independence
• affine recurrences

Innovation

• Neural Finite-State Machine
• Parallel Scan
• Length-Independent State Tracking
• Finite Precision
• Nonaffine Recurrence