ClimateBench v2.0: Probabilistic Climate Model Benchmarking

2026-10-03

AI Summary

This study addresses the absence of unified evaluation criteria for climate models projecting regional temperature and humidity in the 2050s by proposing a three-tier probabilistic benchmark framework encompassing physical plausibility, observational fidelity, and paleoclimate extrapolation. Methodologically, it leverages post-2015 withheld data to achieve genuine out-of-sample validation and employs perfect-model experiments to quantify information value. The assessment integrates multiple dimensions through the Continuous Ranked Probability Score (CRPS), energy conservation diagnostics, and distributional consistency checks. By open-sourcing all code and datasets, this work establishes an equitable comparison platform for multi-paradigm climate models and provides a quantifiable evaluation system for tracking progress in climate projections.

Abstract

We present ClimateBench v2, a standardized protocol for evaluating climate models on diagnostics expected to be informative for their skill in projecting mid-century regional temperature and precipitation changes. The protocol is designed to evaluate any physics-based, data-driven, or hybrid climate model on equal footing using a common set of observational and out-of-distribution tests. We define three tiers of evaluation. Tier I establishes physical credibility through entry-ticket tests of energy conservation, coupled (co-)variability, and basic forced responses. Tier II scores models against post-2015 observations of surface temperature, precipitation, radiative fluxes, sea ice, and key modes of variability using fair CRPS as the primary probabilistic score, complemented by distributional and ensemble-consistency diagnostics. Tier III tests out-of-distribution generalization through paleoclimate simulations spanning the Last Interglacial, Last Glacial Maximum, and Mid-Holocene, and through perfect-model experiments in which data-driven models must predict the future climate of existing Earth system models from historical data alone. We reserve all observational data after 2015 for testing, and submissions must include multiple ensemble members to enable probabilistic evaluation. This reservation exploits a new opportunity provided by the decade of observations accumulated since the end of the CMIP6 historical experiment, which constitutes an out-of-sample record of forced climate change (and internal variability) for the current generation of models, and we quantify, in an idealized setting, the information it carries about mid-century warming. We provide the evaluation code, observational reference datasets, and perfect-model training data as an open benchmark to drive measurable progress in climate projection across all modeling approaches.

Problem

• climate model evaluation
• benchmarking
• probabilistic forecasting
• out-of-distribution generalization
• temperature and precipitation projection

Innovation

• Probabilistic Evaluation
• Out-of-Distribution Generalization
• Climate Model Benchmarking
• Data-driven Climate Models
• CRPS