🤖 AI Summary
This study investigates how recurrent neural networks (RNNs) preserve the topological structure of invariant manifolds—such as those on tori or circles—in regular dynamical systems when trained for time-series prediction. By treating the invariant manifold of the input system as a driving signal, the work posits that the RNN’s hidden state encodes a finite history window rather than instantaneous inputs. A unified theoretical framework is developed by integrating generalized synchronization theory, differential embedding theorems, and contraction analysis. The analysis reveals that under regular dynamical driving, RNNs naturally satisfy smooth embedding conditions, circumventing the stringent requirements typical in chaotic systems. Furthermore, verifiable criteria are established that clarify the relationship between the dimensionality of the hidden state and the intrinsic dimension of the driving system, thereby elucidating the mechanism underlying topologically faithful representations.
📝 Abstract
Recurrent neural networks trained for time-series prediction often develop latent trajectories that preserve qualitative structure of the dynamical systems generating their inputs. Recent empirical work has documented topology-preserving latent organization in trained recurrent models, and recent theoretical results in reservoir computing establish conditions under which the synchronization map is an embedding. Here we synthesize these threads into a unified account of when contracting recurrent networks yield smooth, topology-preserving internal representations for a broad and biologically relevant class of inputs: regular dynamics on invariant circles and tori. Our contribution is an integrated framework that assembles (i) generalized synchronization and embedding guarantees for contracting reservoirs, (ii) regularity mechanisms ensuring differentiability of the synchronization map under mild constraints, and (iii) a base-system viewpoint in which the invariant manifold generating the input stream is treated as the driving system. In this regular setting, the conditions commonly viewed as restrictive in chaotic-attractor analyses become mild and readily satisfied by standard contractive architectures. The framework clarifies how representational content in recurrent circuits is inherently historical: the network state encodes finite windows of input history rather than instantaneous stimuli. By consolidating disparate empirical and theoretical results under common assumptions, the synthesis yields concrete, testable expectations about when prediction-trained recurrent circuits should (or should not) form smooth latent embeddings and how required state dimension scales with the intrinsic dimension of the driving dynamics.