Zero-shot Imputation with Foundation Inference Models for Dynamical Systems

2024-02-12

AI Summary

This work addresses the zero-shot imputation of missing values in time series generated by ordinary differential equation (ODE)-driven dynamical systems—without access to target-domain data for fine-tuning. We propose the first general-purpose framework that pretrains a neural identifier on large-scale synthetic ODE solutions and their sparse, noisy observations, enabling joint amortized inference of initial conditions and state derivatives, followed by numerical integration to reconstruct full trajectories. The method integrates amortized variational inference with neural operators to establish a probabilistic modeling and generalization architecture tailored to the ODE solution manifold. Evaluated across 63 diverse synthetic dynamical systems and 10 high-dimensional real-world benchmarks—including human motion capture, air quality monitoring, and Navier–Stokes simulations—our approach achieves state-of-the-art zero-shot performance, consistently outperforming task-specific fine-tuned methods. This establishes a new paradigm for zero-shot time-series imputation in ODE-driven systems.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Dynamical systems governed by ordinary differential equations (ODEs) serve as models for a vast number of natural and social phenomena. In this work, we offer a fresh perspective on the classical problem of imputing missing time series data, whose underlying dynamics are assumed to be determined by ODEs. Specifically, we revisit ideas from amortized inference and neural operators, and propose a novel supervised learning framework for zero-shot time series imputation, through parametric functions satisfying some (hidden) ODEs. Our proposal consists of two components. First, a broad probability distribution over the space of ODE solutions, observation times and noise mechanisms, with which we generate a large, synthetic dataset of (hidden) ODE solutions, along with their noisy and sparse observations. Second, a neural recognition model that is trained offline, to map the generated time series onto the spaces of initial conditions and time derivatives of the (hidden) ODE solutions, which we then integrate to impute the missing data. We empirically demonstrate that one and the same (pretrained) recognition model can perform zero-shot imputation across 63 distinct time series with missing values, each sampled from widely different dynamical systems. Likewise, we demonstrate that it can perform zero-shot imputation of missing high-dimensional data in 10 vastly different settings, spanning human motion, air quality, traffic and electricity studies, as well as Navier-Stokes simulations -- without requiring any fine-tuning. What is more, our proposal often outperforms state-of-the-art methods, which are trained on the target datasets. Our pretrained model, repository and tutorials are available online.

Problem

• Imputing missing time series data governed by ODEs.
• Zero-shot imputation across diverse dynamical systems.
• High-dimensional data imputation without fine-tuning.

Innovation

• Zero-shot imputation using neural recognition models
• Synthetic dataset generation for ODE solutions
• Pretrained model for diverse dynamical systems