Structural Limits of the Information-Theoretic Uncertainty Decomposition

2026-09-30

AI Summary

This study addresses the entanglement of aleatoric and epistemic uncertainty in machine learning uncertainty estimation, along with the phenomenon of "epistemic collapse." By analyzing information-theoretic decomposition frameworks from a function-space perspective and integrating Monte Carlo sampling theory, this work investigates the probabilistic distribution properties of ensemble models. It provides the first theoretical quantification of the upper bound on the infeasible region under finite-sample conditions (AU ≤ log(2)/N), revealing the structural origins of epistemic collapse. Furthermore, it demonstrates that increasing ensemble size effectively mitigates this issue, while cautioning against the risk of strong coupling between the two uncertainty types under low aleatoric uncertainty regimes.

Abstract

Uncertainty estimation in machine learning typically decomposes uncertainty into aleatoric uncertainty (AU) and epistemic uncertainty (EU) using the standard information-theoretic framework. However, in practice, two critical issues arise: entanglement (AU and EU are highly correlated) and epistemic collapse (EU magnitude shrinks with increasing model capacity). We analyze this framework on a functional level and discover that significant portions of the assumed AU, EU range are infeasible in finite settings, and cannot be attained with any class probabilities. We characterize how this infeasible region scales with the number of classes and Monte Carlo samples $N$ (e.g., from ensembles with $N$ members), revealing it is bounded by $\text{AU} \leq \log(2)/N$. Crucially, the infeasible region's boundary helps explain epistemic collapse: when model confidence is high, $\text{AU} > \text{EU}$ is guaranteed by this fundamental structural limitation. Our findings show that increasing ensemble size mitigates epistemic collapse by reducing the infeasible area. Lastly, we caution against interpreting AU and EU as independent quantities in low AU regimes, since we show they are coupled when $\text{AU} \leq \log(2)/N$.

Problem

• uncertainty decomposition
• aleatoric uncertainty
• epistemic uncertainty
• epistemic collapse
• entanglement

Innovation

• Uncertainty Decomposition
• Epistemic Collapse
• Aleatoric Uncertainty
• Information Theory
• Ensemble Methods