PhysMetrics.Weather: An Evaluation Framework for Physical Consistency in ML Weather Models

2026-06-09

AI Summary

This work addresses the lack of physical consistency guarantees in current machine learning–based weather forecasting models and their overreliance on data-driven evaluation metrics. To bridge this gap, we propose the first multidimensional framework for assessing the physical fidelity of learned weather models, systematically quantifying consistency across three key dimensions: conservation laws, spectral characteristics, and dynamical behavior. The framework integrates techniques such as conservation law verification, spectral analysis, and dynamical consistency checks, and is implemented in an open-source, reproducible toolkit named PhysMetrics.Weather. By providing standardized, physics-aware evaluation protocols, our approach fills a critical void in benchmarking standards for physically informed weather models and supports both their development and operational reliability validation.

Abstract

Machine learning weather prediction (MLWP) models have achieved impressive forecasting performance at a small fraction of the computational costs required for traditional physics-based methods. However, they are primarily (1) data-driven and (2) evaluated using pixel-wide error metrics (e.g., RMSE), so there are no guarantees that their forecasts are consistent with known physical laws. We introduce PhysMetrics.Weather, an evaluation framework that assesses the physical realism of MLWP models across three types of metrics: conservation, spectral, and dynamical. By quantifying physical realism, this tool guides the development of physics-informed architectures and helps evaluate whether MLWP models are reliable for operational use. Our framework is available on Github at https://github.com/Emmakast/PhysMetrics.Weather.

Problem

• machine learning weather prediction
• physical consistency
• evaluation framework
• physics-based methods
• forecast reliability

Innovation

• physical consistency
• evaluation framework
• machine learning weather prediction
• physics-informed ML
• conservation laws