Liquid Resistance Liquid Capacitance Networks

2024-01-30

AI Summary

Existing electro-equivalent circuits (EECs), liquid time-constant networks (LTCs), and their saturated variants suffer from limited generalizability, accuracy, and biological interpretability; meanwhile, gated RNNs face inefficiency and non-differentiability issues. To address these limitations, this paper proposes liquid resistance–capacitance networks (LRCs), a novel neural differential equation model that integrates circuit-theoretic priors with liquid time-constant dynamics. LRCs introduce a first-of-its-kind *liquid capacitance* mechanism to suppress oscillations, enhance stability, and improve modeling fidelity. We further derive the lightweight LRC unit (LRCU), which achieves high-accuracy, differentiable, and interpretable temporal modeling via a single-step explicit Euler discretization. Evaluated on multiple time-series benchmarks and an autonomous driving lane-keeping task, LRCs/LRCU consistently outperform state-of-the-art neural ODEs and gated RNNs in prediction accuracy, computational efficiency, and neurodynamical interpretability.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Cognitive Modeling & Cognitive Systems: Neural Spike CodingNatural Language Processing: (Large) Language Models

Application Category

Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphsSearch and Retrieval-Augmented AI: Large language models for searchEconomics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

We introduce liquid-resistance liquid-capacitance neural networks (LRCs), a neural-ODE model which considerably improve the generalization, accuracy, and biological plausibility of electrical equivalent circuits (EECs), liquid time-constant networks (LTCs), and saturated liquid time-constant networks (STCs), respectively. We also introduce LRC units (LRCUs), as a very efficient and accurate gated RNN-model, which results from solving LRCs with an explicit Euler scheme using just one unfolding. We empirically show and formally prove that the liquid capacitance of LRCs considerably dampens the oscillations of LTCs and STCs, while at the same time dramatically increasing accuracy even for cheap solvers. We experimentally demonstrate that LRCs are a highly competitive alternative to popular neural ODEs and gated RNNs in terms of accuracy, efficiency, and interpretability, on classic time-series benchmarks and a complex autonomous-driving lane-keeping task.

Problem

• Improving generalization and accuracy of neural-ODE models
• Reducing oscillations in liquid time-constant networks
• Enhancing efficiency and interpretability of gated RNNs

Innovation

• Liquid-resistance liquid-capacitance neural networks (LRCs)
• LRC units (LRCUs) as efficient gated RNN-model
• Liquid capacitance dampens oscillations, increases accuracy