Technology Category
Machine Learning: Calibration & Uncertainty QuantificationReasoning under Uncertainty: Graphical ModelsIntelligent Robots: State Estimation
Partitioning the Sample Space for a More Precise Shannon Entropy Estimation
🤖 AI Summary
Reliable estimation of Shannon entropy from small samples suffers from systematic underestimation, particularly when the sample size is smaller than the number of possible outcomes. To address this, we propose a novel discrete entropy estimator that jointly models sample-space partitioning, missing-mass estimation, and unseen-outcome count estimation—leveraging entropy’s decomposability, interpolation-based estimation, empirical frequency correction, and probabilistic modeling of unseen events for synergistic bias correction. The method substantially mitigates negative bias and significantly outperforms classical estimators—including maximum likelihood estimation (MLE) and jackknife—in the undersampled regime. Its performance matches that of state-of-the-art approaches while exhibiting superior robustness and higher accuracy across diverse distributions and sampling conditions.
Technology Category
Application Category
📝 Abstract
Reliable data-driven estimation of Shannon entropy from small data sets, where the number of examples is potentially smaller than the number of possible outcomes, is a critical matter in several applications. In this paper, we introduce a discrete entropy estimator, where we use the decomposability property in combination with estimations of the missing mass and the number of unseen outcomes to compensate for the negative bias induced by them. Experimental results show that the proposed method outperforms some classical estimators in undersampled regimes, and performs comparably with some well-established state-of-the-art estimators.
Problem
Research questions and friction points this paper is trying to address.
Estimates Shannon entropy from small datasets
Corrects bias from missing mass and unseen outcomes
Improves accuracy in undersampled regimes
Innovation
Methods, ideas, or system contributions that make the work stand out.
Uses decomposability property for entropy estimation
Estimates missing mass and unseen outcomes
Compensates negative bias in undersampled regimes
🔎 Similar Papers
No similar papers found.
