🤖 AI Summary
Current evaluation methods for probabilistic forecasts rely on single scalar metrics, which fail to reveal the trade-off between sharpness and accuracy. This work proposes the Interval Score ROC curve (IS-ROC), the first geometric representation that fully characterizes families of interval forecasts across varying levels of sharpness while guaranteeing Pareto optimality and convexity. Leveraging the geometric properties of the IS-ROC, the authors further develop a tangent-based optimization method for calibration and a convex hull ensemble strategy. Experimental results demonstrate that this framework significantly outperforms existing approaches in terms of evaluation comprehensiveness, calibration quality, and ensemble performance.
📝 Abstract
Probabilistic forecast evaluation is inherently multi-objective, yet existing proper scoring rules reduce predictive performance to a single scalar value, potentially obscuring the trade-off between forecast concentration and predictive accuracy. We introduce the Interval-Score Receiver Operating Characteristic (IS-ROC) Curve, a graphical framework that represents the complete family of interval forecasts generated by varying prediction tightness. We show that the IS-ROC Curve induced by the data generating process is Pareto optimal and convex, providing a geometric characterization of the optimal forecasting frontier. Building on these properties, we propose a geometry-based calibration procedure based on tangent optimization and convexification, together with an ensemble strategy that combines competing forecasters through convex hull construction. Finally, we provide a practical workflow and numerical examples illustrating forecast comparison, calibration, and ensemble construction within the proposed framework.