Zero-shot forecasting of chaotic systems

2024-09-24

arXiv.org

AI Summary

This paper investigates the zero-shot time-series forecasting capability of large time-series foundation models (TSMs) on chaotic systems. Methodologically, it leverages multi-source, large-scale pre-trained TSMs combined with in-context learning—requiring no fine-tuning—to predict trajectories across 135 distinct chaotic dynamical systems from minimal contextual examples. The contributions are threefold: (i) the first empirical demonstration that TSMs can preserve the geometric structure and statistical properties of chaotic attractors (e.g., Lyapunov exponents, fractal dimension) in zero-shot settings; (ii) identification of context rephrasing as a critical mechanism enabling long-horizon chaotic behavior capture; and (iii) competitive point-prediction accuracy against specialized models (e.g., NBEATS, TiDE) under data scarcity, with robust retention of intrinsic dynamical structure—even when point forecasts fail.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Other Foundations of Cognitive Modeling & Systems

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Time-series forecasting is a challenging problem that traditionally requires specialized models custom-trained for the specific task at hand. Recently, inspired by the success of large language models, foundation models pre-trained on vast amounts of time-series data from diverse domains have emerged as a promising candidate for general-purpose time-series forecasting. The defining characteristic of these foundation models is their ability to perform zero-shot learning, that is, forecasting a new system from limited context data without explicit re-training or fine-tuning. Here, we evaluate whether the zero-shot learning paradigm extends to the challenging task of forecasting chaotic systems. Across 135 distinct chaotic dynamical systems and $10^8$ timepoints, we find that foundation models produce competitive forecasts compared to custom-trained models (including NBEATS, TiDE, etc.), particularly when training data is limited. Interestingly, even after point forecasts fail, large foundation models are able to preserve the geometric and statistical properties of the chaotic attractors. We attribute this success to foundation models' ability to perform in-context learning and identify context parroting as a simple mechanism used by these models to capture the long-term behavior of chaotic dynamical systems. Our results highlight the potential of foundation models as a tool for probing nonlinear and complex systems.

Problem

• Zero-shot forecasting of chaotic systems without retraining.
• Evaluating foundation models' performance on chaotic dynamics.
• Preserving chaotic attractors' properties despite forecast failures.

Innovation

• Zero-shot learning for chaotic systems forecasting
• Foundation models outperform custom-trained models
• Preserves chaotic attractors' geometric and statistical properties