Reliability of Probabilistic Emulation of Physical Systems

2026-06-11

AI Summary

This study addresses the lack of systematic evaluation of uncertainty reliability in probabilistic forecasting for physical systems, particularly between generative models and ensemble methods trained with the Continuous Ranked Probability Score (CRPS). The authors establish a unified evaluation framework to conduct the first systematic comparison of these two approaches under identical model scales and computational budgets in two-dimensional spatiotemporal systems. Results demonstrate that CRPS-based ensembles consistently achieve more reliable uncertainty coverage and faster inference in both single-step and roll-out predictions. Generative models exhibit comparable coverage only when trained in the original data space but incur significantly higher latency. Both approaches yield similar predictive accuracy. Notably, CRPS ensembles maintain strong performance even in latent spaces, highlighting their advantage in enabling efficient and reliable uncertainty quantification.

Abstract

Two dominant approaches have emerged for generating probabilistic forecasts of physical systems: generative models, such as diffusion or flow matching; and ensembles of deterministic models with stochasticity injected, trained using the continuous ranked probability score (CRPS) loss. While both approaches have demonstrated strong predictive accuracy, the reliability of their uncertainties has not been systematically assessed. We address this gap by developing a framework to evaluate both approaches across diverse 2D spatiotemporal physical systems, under matched model size and computational budget. We assess the reliability of probabilistic emulation by inspecting the empirical coverage of predictive intervals, while also considering accuracy and computational efficiency metrics. CRPS-trained ensembles typically achieve more reliable uncertainties on both single-step prediction and autoregressive rollouts, demonstrating better coverage than the standard alternative of training generative models in a latent space. Moreover, the CRPS approach offers significantly faster inference. When generative models are trained in ambient rather than a compressed latent space, which is often infeasible for high-dimensional problems, they exhibit comparable coverage to CRPS-trained ensembles, though with substantially larger inference latency. In contrast, when CRPS-trained ensembles are trained in latent space they do not show a marked degradation in coverage with respect to ambient space. Both generative models and CRPS-trained ensembles demonstrate good predictive accuracy. To facilitate future research and application, we release AutoCast, a modular framework implementing both generative models and CRPS-trained ensembles, alongside AutoSim, a flexible dataset generation package for rapid prototyping.

Problem

• probabilistic forecasting
• uncertainty reliability
• physical systems
• generative models
• CRPS-trained ensembles

Innovation

• probabilistic emulation
• CRPS-trained ensembles
• generative models
• uncertainty reliability
• spatiotemporal forecasting