Possible, Yes; Ignorant, Perhaps: A Scorecard for Possibilistic Forecasts

2026-04-02

AI Summary

This study addresses the limitation of traditional probabilistic forecasting in expressing epistemic uncertainty—specifically, the inability to convey “I don’t know”—due to normalization constraints. By leveraging possibility theory, the authors propose a novel framework that explicitly models ignorance as the non-normalization of possibility distributions, integrating perspectives from possibility, probability, and classification. The framework enables fine-grained diagnosis of forecast failure modes through a five-dimensional scoring system—assessing validity, sharpness, ignorance, dominance, and an overall composite metric—augmented with operational metrics such as POD, FAR, and CSI. Experiments on Storm Prediction Center convective outlook data demonstrate that explicitly representing ignorance yields superior performance compared to enforcing normalization, and synthetic reforecast experiments validate both the method’s efficacy and its interpretability through visualization.

Technology Category

Reasoning under Uncertainty: Other Foundations of Reasoning under UncertaintyMachine Learning: Calibration & Uncertainty QuantificationKnowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsGraph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphsUser Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Probabilistic forecasts must sum to unity and cannot express ``I don't know.'' Possibility theory relaxes this constraint: a subnormal distribution explicitly measures how much of the plausibility budget remains unassigned, ignorance signal that probability cannot represent. This paper develops a verification framework for such forecasts, centred on a five-number scorecard that separately diagnoses whether the forecast pointed at the right outcome (depth-of-truth), how sharply (diffuseness, support margin), how confidently (ignorance), and how dominantly (conditional necessity). A possibility-to-probability conversion preserves ignorance for familiar frequency-based scoring; categorical threshold scores (POD, FAR, CSI, etc.) connect to operational practice. Together, these three complementary facets -- possibilistic, probabilistic, and categorical -- expose failure modes invisible to any single metric. Storm Prediction Center convective outlook categories serve as the running example throughout; a synthetic reforecast demonstrates diagnostic visualisations and scorecard interpretation. Ignorance is better expressed than repressed.

Problem

• possibilistic forecasts
• ignorance
• forecast verification
• uncertainty representation
• scorecard

Innovation

• possibility theory
• forecast verification
• ignorance representation
• scorecard framework
• possibilistic forecasting