Information leakage from data revisions in retrospective forecasts

2026-08-06

AI Summary

This study addresses a critical yet often overlooked source of information leakage in retrospective evaluations of AI-driven forecasting systems: the failure to account for data revisions. The authors systematically demonstrate how this oversight leads to inflated performance estimates and present, for the first time, a cross-domain cautionary framework to mitigate such biases. Through retrospective predictive analysis, explicit modeling of data revision processes, and rigorous evaluation protocols, they reveal that a previously reported AI system’s purported superiority over the CDC ensemble model stems not from genuine predictive gains but from information leakage introduced by unadjusted historical data revisions. These findings establish essential methodological corrections and evaluation standards for future research in AI-based forecasting, ensuring more reliable and reproducible assessments of predictive performance.

Abstract

Aygün et al (2026, https://doi.org/10.1038/s41586-026-10658-6) claim that their AI-driven Empirical Research Assistance (ERA) system produces COVID-19 hospitalisation forecasts which outperform the state-of-the-art CDC ensemble by a considerable margin for the 2024/25 season. We demonstrate that the observed performance gain is attributable to information leakage in the retrospective forecasting setup, which resulted because data revisions were not taken into account. As similar mechanisms are at play in many other forecasting fields, our cautionary tale applies not just to epidemic forecasting, but is relevant to the entire emerging field of AI-assisted predictive modelling.

Problem

• information leakage
• data revisions
• retrospective forecasting
• AI-assisted predictive modelling
• epidemic forecasting

Innovation

• information leakage
• data revisions
• retrospective forecasting
• AI-assisted predictive modelling
• epidemic forecasting