🤖 AI Summary
This paper addresses high-probability estimation of discrete distributions over a finite alphabet under the Kullback–Leibler (KL) divergence, focusing on the sparse regime where the sample size is smaller than the alphabet size. We propose a data-driven, adaptive smoothing estimator tailored to this setting. Our main contributions are threefold: First, we establish the first non-asymptotic, high-probability upper bound on the KL risk, which is minimax optimal up to constant factors. Second, we derive a sharp high-probability upper bound on the missing mass. Third, we prove that the KL risk of the Laplace estimator admits tight high-probability upper and lower bounds; that the minimax high-probability risk incurs an extra logarithmic factor; and—under a sparsity assumption—that the risk bound depends only on two effective sparsity parameters, enabling automatic adaptation to underlying distribution structure. Collectively, these results unify and substantially extend the theoretical foundations of classical smoothing methods, delivering rigorous, practical risk guarantees for small-sample discrete distribution estimation.
📝 Abstract
We study the problem of estimating a distribution over a finite alphabet from an i.i.d. sample, with accuracy measured in relative entropy (Kullback-Leibler divergence). While optimal expected risk bounds are known, high-probability guarantees remain less well-understood. First, we analyze the classical Laplace (add-$1$) estimator, obtaining matching upper and lower bounds on its performance and showing its optimality among confidence-independent estimators. We then characterize the minimax-optimal high-probability risk achievable by any estimator, which is attained via a simple confidence-dependent smoothing technique. Interestingly, the optimal non-asymptotic risk contains an additional logarithmic factor over the ideal asymptotic risk. Next, motivated by scenarios where the alphabet exceeds the sample size, we investigate methods that adapt to the sparsity of the distribution at hand. We introduce an estimator using data-dependent smoothing, for which we establish a high-probability risk bound depending on two effective sparsity parameters. As part of the analysis, we also derive a sharp high-probability upper bound on the missing mass.