Epistemic Learning from Imprecise Annotation

2026-09-28

AI Summary

This study addresses the limitation that a single predictive distribution under imprecise annotations obscures evidential uncertainty and fails to capture annotation ambiguity. To this end, it proposes POCC, an epistemic learning framework based on credal set supervision. This framework uniquely employs convex sets as supervisory signals, efficiently deriving credal sets via label relaxation. It utilizes a shared backbone with a pessimistic-optimistic dual-head architecture, trained through closed-form cross-entropy inner optimization and label smoothing to generate predictive credal sets for quantifying epistemic uncertainty. Theoretically, the work establishes finite-sample generalization bounds incorporating imprecision penalties. Empirically, POCC outperforms baseline methods in both human disagreement and teacher prediction scenarios, achieving a favorable trade-off among accuracy, calibration, and selective classification.

Abstract

Imprecise annotations may support several plausible labelling distributions, yet learning methods often resolve this ambiguity into a single predictive distribution. This can obscure what the annotation evidence leaves unresolved. We introduce epistemic learning from credal supervision, a framework that uses convex sets of plausible labelling distributions, called credal sets, as supervision and learns sets of predictive distributions. We instantiate the framework with the pessimistic--optimistic credal classifier (POCC), which combines a shared backbone with two classification heads trained to minimise worst-case and best-case losses over the supervision sets. Their outputs define a predictive credal set whose spread provides an uncertainty score. We also show how credal labels can be obtained through a simple relaxation of existing probabilistic labels, reducing commitment to their precise probability assignments. This construction admits closed-form inner optimisation under cross-entropy loss, enabling efficient training. Assuming the supervision sets contain the true conditional label distributions, and other regularity assumptions, we establish a finite-sample generalisation bound for the averaged predictor with an explicit penalty for supervision imprecision. We evaluate POCC using human annotator disagreement and teacher predictions, alongside label smoothing as a controlled proxy for annotation imprecision. Across these settings, POCC achieves a favourable balance of predictive accuracy, calibration, and uncertainty-based selective classification versus competitive baselines.

Problem

• imprecise annotation
• epistemic uncertainty
• credal sets
• ambiguity
• predictive distribution

Innovation

• Credal Sets
• Epistemic Learning
• Imprecise Annotation
• POCC
• Generalisation Bound