Oscillatory State-Space Models

2024-10-04

arXiv.org

AI Summary

Addressing the challenge of balancing long-term prediction stability and computational efficiency in modeling long sequences, this paper proposes the Linear Oscillatory State Space (LinOSS) model. LinOSS is grounded in the forced harmonic oscillator differential equation and employs a nonnegative diagonal state matrix to ensure lightweight, intrinsic stability. It leverages implicit-explicit discretization and a fast associative parallel scan algorithm to enable efficient, scalable sequence processing. Theoretically, LinOSS is proven to possess universal function approximation capability and preserve time-reversal symmetry. In long-range forecasting tasks up to 50k steps, LinOSS achieves nearly 2× higher accuracy than Mamba and 2.5× higher than LRU, significantly outperforming existing state-of-the-art methods.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We propose Linear Oscillatory State-Space models (LinOSS) for efficiently learning on long sequences. Inspired by cortical dynamics of biological neural networks, we base our proposed LinOSS model on a system of forced harmonic oscillators. A stable discretization, integrated over time using fast associative parallel scans, yields the proposed state-space model. We prove that LinOSS produces stable dynamics only requiring nonnegative diagonal state matrix. This is in stark contrast to many previous state-space models relying heavily on restrictive parameterizations. Moreover, we rigorously show that LinOSS is universal, i.e., it can approximate any continuous and causal operator mapping between time-varying functions, to desired accuracy. In addition, we show that an implicit-explicit discretization of LinOSS perfectly conserves the symmetry of time reversibility of the underlying dynamics. Together, these properties enable efficient modeling of long-range interactions, while ensuring stable and accurate long-horizon forecasting. Finally, our empirical results, spanning a wide range of time-series tasks from mid-range to very long-range classification and regression, as well as long-horizon forecasting, demonstrate that our proposed LinOSS model consistently outperforms state-of-the-art sequence models. Notably, LinOSS outperforms Mamba by nearly 2x and LRU by 2.5x on a sequence modeling task with sequences of length 50k.

Problem

• Efficient learning on long sequences
• Stable and accurate long-horizon forecasting
• Outperforming state-of-the-art sequence models

Innovation

• Linear Oscillatory State-Space models
• Forced harmonic oscillators system
• Fast associative parallel scans