Fixing the Double Penalty in Data-Driven Weather Forecasting Through a Modified Spherical Harmonic Loss Function

2025-01-31

AI Summary

Existing weather forecasting models suffer from the “double-penalty” effect inherent in mean-squared-error loss, limiting their ability to accurately represent small-scale dynamics and extreme events. To address this, we propose a parameter-free spherical harmonic loss function—the first to incorporate spherical harmonic decomposition into loss design—thereby decoupling correlation degradation from spectral amplitude errors and eliminating the need for hyperparameter tuning. Our method integrates spherical harmonic transforms, spectral loss modeling, GraphCast fine-tuning, and joint optimization of deterministic and ensemble forecasts. Experiments demonstrate that forecast effective resolution improves from 1250 km to 160 km. Significant gains are achieved in predicting tropical cyclone intensity and surface wind extremes, while ensemble spread is better calibrated, enhancing forecast reliability and probabilistic skill.

Technology Category

• Machine Learning: Ensemble Methods
• Planning, Routing, and Scheduling: Model-Based Reasoning
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent advancements in data-driven weather forecasting models have delivered deterministic models that outperform the leading operational forecast systems based on traditional, physics-based models. However, these data-driven models are typically trained with a mean squared error loss function, which causes smoothing of fine scales through a"double penalty"effect. We develop a simple, parameter-free modification to this loss function that avoids this problem by separating the loss attributable to decorrelation from the loss attributable to spectral amplitude errors. Fine-tuning the GraphCast model with this new loss function results in sharp deterministic weather forecasts, an increase of the model's effective resolution from 1,250km to 160km, improvements to ensemble spread, and improvements to predictions of tropical cyclone strength and surface wind extremes.

Problem

• Weather Prediction
• Model Limitations
• Accuracy Improvement

Innovation

• Improved Spherical Harmonics
• Enhanced Weather Prediction
• Small-scale Weather Detail