Do machine learning climate models work in changing climate dynamics?

2025-09-15

AI Summary

Accelerating climate change is increasing the frequency and severity of extreme events, causing them to deviate significantly from historical distributions (out-of-distribution, OOD). While modern machine learning (ML) climate models achieve high accuracy and efficiency, their generalization performance under distributional shifts remains poorly understood. Method: This work introduces, for the first time, a systematic OOD evaluation paradigm to climate forecasting. We propose a robustness assessment framework tailored to climate models, integrating state-of-the-art ML architectures, multi-source observational and reanalysis climate data, and advanced OOD detection techniques to quantitatively evaluate predictive performance across diverse climate shift scenarios. Contribution/Results: Experiments reveal severe limitations in current models’ OOD generalization—marked by substantial performance degradation and high variance across extreme-event regimes. Model architecture critically influences robustness, with notable inter-architectural disparities. We identify key failure modes under distributional shift, establishing foundational theoretical insights and methodological tools for trustworthy climate risk prediction.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Adversarial Attacks & Robustness
• Data Mining & Knowledge Management: Anomaly/Outlier Detection

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Climate change is accelerating the frequency and severity of unprecedented events, deviating from established patterns. Predicting these out-of-distribution (OOD) events is critical for assessing risks and guiding climate adaptation. While machine learning (ML) models have shown promise in providing precise, high-speed climate predictions, their ability to generalize under distribution shifts remains a significant limitation that has been underexplored in climate contexts. This research systematically evaluates state-of-the-art ML-based climate models in diverse OOD scenarios by adapting established OOD evaluation methodologies to climate data. Experiments on large-scale datasets reveal notable performance variability across scenarios, shedding light on the strengths and limitations of current models. These findings underscore the importance of robust evaluation frameworks and provide actionable insights to guide the reliable application of ML for climate risk forecasting.

Problem

• Evaluating ML climate models' generalization under distribution shifts
• Assessing performance variability in out-of-distribution climate scenarios
• Testing ML reliability for climate risk forecasting in changing dynamics

Innovation

• Adapting OOD evaluation methodologies to climate data
• Systematically evaluating ML models in OOD scenarios
• Providing robust evaluation frameworks for climate forecasting