Dynamical errors in machine learning forecasts

2025-04-15

AI Summary

Standard error metrics (e.g., MAE, MSE) fail to capture dynamical consistency in machine learning predictions—a critical limitation for physical forecasting. This work addresses the frequent distortion of intrinsic dynamical properties—such as chaotic structure and persistence—in long-horizon or recursive predictions. To this end, we propose a novel paradigm for evaluating physical fidelity by introducing two interpretable dynamical fidelity indicators: instantaneous dimension $d$ and inverse persistence $ heta$. We formulate a new error measure grounded in these quantities, revealing an intrinsic relationship between prediction error and system complexity/non-persistence. Our method integrates robust $d$ estimation with $ heta$ computation and is compatible with both direct and recursive prediction frameworks. Extensive validation is conducted on canonical systems—including the Lorenz system, Kuramoto–Sivashinsky equation, Kolmogorov flow—as well as real-world weather data. Results demonstrate that the proposed metrics provide essential diagnostic insights beyond standard errors, enabling principled assessment and improvement of dynamical behavior in learned models.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Intelligent Robots: State Estimation
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements

Abstract

In machine learning forecasting, standard error metrics such as mean absolute error (MAE) and mean squared error (MSE) quantify discrepancies between predictions and target values. However, these metrics do not directly evaluate the physical and/or dynamical consistency of forecasts, an increasingly critical concern in scientific and engineering applications. Indeed, a fundamental yet often overlooked question is whether machine learning forecasts preserve the dynamical behavior of the underlying system. Addressing this issue is essential for assessing the fidelity of machine learning models and identifying potential failure modes, particularly in applications where maintaining correct dynamical behavior is crucial. In this work, we investigate the relationship between standard forecasting error metrics, such as MAE and MSE, and the dynamical properties of the underlying system. To achieve this goal, we use two recently developed dynamical indices: the instantaneous dimension ($d$), and the inverse persistence ($ heta$). Our results indicate that larger forecast errors -- e.g., higher MSE -- tend to occur in states with higher $d$ (higher complexity) and higher $ heta$ (lower persistence). To further assess dynamical consistency, we propose error metrics based on the dynamical indices that measure the discrepancy of the forecasted $d$ and $ heta$ versus their correct values. Leveraging these dynamical indices-based metrics, we analyze direct and recursive forecasting strategies for three canonical datasets -- Lorenz, Kuramoto-Sivashinsky equation, and Kolmogorov flow -- as well as a real-world weather forecasting task. Our findings reveal substantial distortions in dynamical properties in ML forecasts, especially for long forecast lead times or long recursive simulations, providing complementary information on ML forecast fidelity that can be used to improve ML models.

Problem

• Assessing dynamical consistency in ML forecasts
• Relating standard errors to system dynamics
• Proposing dynamical indices-based error metrics

Innovation

• Uses dynamical indices d and θ
• Proposes error metrics for dynamical consistency
• Analyzes ML forecasts with canonical datasets