Continuous-Time Piecewise-Linear Recurrent Neural Networks

2026-02-17

AI Summary

Existing piecewise-linear recurrent neural networks (PLRNNs) are limited to discrete time and struggle to model continuous-time, irregularly sampled physical and biological systems. This work proposes a continuous-time PLRNN (cPLRNN), establishing its theoretical foundation for the first time. By leveraging ReLU-based piecewise-linear dynamics, the cPLRNN enables efficient simulation and training without numerical integration and supports semi-analytical analysis of dynamical structures such as fixed points and limit cycles. In reconstructing dynamical systems characterized by discontinuities and hard thresholds, the cPLRNN substantially outperforms both discrete-time PLRNNs and neural ordinary differential equations, achieving high accuracy while offering enhanced mechanistic interpretability.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Planning, Routing, and Scheduling: Mixed Discrete/Continuous Planning
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

In dynamical systems reconstruction (DSR) we aim to recover the dynamical system (DS) underlying observed time series. Specifically, we aim to learn a generative surrogate model which approximates the underlying, data-generating DS, and recreates its long-term properties (`climate statistics'). In scientific and medical areas, in particular, these models need to be mechanistically tractable -- through their mathematical analysis we would like to obtain insight into the recovered system's workings. Piecewise-linear (PL), ReLU-based RNNs (PLRNNs) have a strong track-record in this regard, representing SOTA DSR models while allowing mathematical insight by virtue of their PL design. However, all current PLRNN variants are discrete-time maps. This is in disaccord with the assumed continuous-time nature of most physical and biological processes, and makes it hard to accommodate data arriving at irregular temporal intervals. Neural ODEs are one solution, but they do not reach the DSR performance of PLRNNs and often lack their tractability. Here we develop theory for continuous-time PLRNNs (cPLRNNs): We present a novel algorithm for training and simulating such models, bypassing numerical integration by efficiently exploiting their PL structure. We further demonstrate how important topological objects like equilibria or limit cycles can be determined semi-analytically in trained models. We compare cPLRNNs to both their discrete-time cousins as well as Neural ODEs on DSR benchmarks, including systems with discontinuities which come with hard thresholds.

Problem

• dynamical systems reconstruction
• continuous-time modeling
• piecewise-linear RNNs
• irregular time series
• model tractability

Innovation

• continuous-time PLRNN
• dynamical systems reconstruction
• mechanistic interpretability
• semi-analytical topology
• irregular time series