🤖 AI Summary
This study addresses the limitation of existing recalibration methods, which often obscure miscalibration in specific regions such as extreme events. To overcome this, we propose an outcome-conditional recalibration post-processing method that leverages quantile recalibration and conditional distribution scaling to achieve precise correction of arbitrary predictive distributions within user-defined regions. By simultaneously preserving global performance and local reliability, the proposed approach significantly enhances conditional calibration on regression benchmark tasks. Furthermore, when applied to electricity price forecasting, it substantially improves calibration in negative-price regimes with negligible accuracy loss. Overall, this work provides more reliable localized guarantees for probabilistic forecasting.
📝 Abstract
Calibration is an essential requirement for probabilistic predictions to be useful for decision making. While state-of-the-art prediction methods often yield miscalibrated predictive distributions, several post-hoc recalibration schemes have been proposed to generate calibrated predictions. However, popular recalibration schemes can conceal miscalibration in specific regions of the outcome space. Since particular outcomes, such as extreme events, often matter most for decision making, probabilistic predictions should be calibrated when evaluation is restricted to these outcomes. Hence, in this paper, we introduce outcome-conditional recalibration, a post-hoc method to recalibrate probabilistic predictions on user-defined regions of the outcome space. The method is simple, easy to implement, and can be applied to arbitrary predictive distributions. It works by applying the quantile recalibration approach of Kuleshov et al. (2018) to forecast conditional distributions, before rescaling these conditional distributions so that forecast event probabilities match empirical occurrence frequencies. This produces valid and continuous predictive distributions that are calibrated within each region of interest. Across regression benchmarks, we demonstrate that existing recalibration schemes do not necessarily yield calibrated predictions when interest is on particular outcomes, and that our approach improves outcome-conditional calibration relative to existing conditional and unconditional recalibration methods, while retaining competitive calibration overall. In an application to day-ahead electricity price forecasting, the approach substantially improves calibration when predicting negative prices, at negligible cost to forecast accuracy.