Settling the Sample Complexity of Rényi Entropy Estimation

2026-10-07

AI Summary

This study addresses the long-standing open problem of the sample complexity for non-integer order Rényi entropy estimation, which has constrained theoretical progress in this domain. For discrete distributions of support size d and any non-integer order α > 0, this work proposes a novel construction technique with bounded higher-order moments to establish tight lower bounds, alongside an improved polynomial approximation estimator that effectively reduces bias. Consequently, it provides the first complete characterization of the sample complexity for this problem, deriving exact Θ-order necessary and sufficient sample size formulas. By bridging this critical theoretical gap, the research refines the theoretical framework of Rényi entropy estimation and establishes a rigorous foundation for the design of efficient algorithms.

Abstract

Rényi entropy estimation has been comprehensively investigated by Acharya, Orlitsky, Suresh and Tyagi (SODA 2015; IEEE Trans. Inf. Theory 2017) and consequent works, whereas only the sample complexity of Rényi entropy estimation of integer order has been settled. In this paper, we settle the sample complexity of Rényi entropy estimation of noninteger order, thereby completing the complexity picture of Rényi entropy estimation. Specifically, we show that for any noninteger $α> 0$, it is sufficient and necessary to use \[ Θ\!\left(\frac{d^{\max\{1/α,1\}}}{\varepsilon^{1/α}\log(d)} + \frac{d^{|1-1/α|}}{\varepsilon^2}\right) \] samples to estimate the Rényi entropy of order $α$ of an unknown discrete distribution over an alphabet of size $d$ to within additive error $\varepsilon$. For the upper bound, we reduce the bias using a refined polynomial approximation estimator for large probabilities. For the lower bound, we employ a different hard instance equipped with a new moment matching construction. The constructive moment matching has constant bounded high-order moments, while attaining a fixed ratio between the $α$-th moments, which is of independent interest.

Problem

• Rényi entropy estimation
• sample complexity
• noninteger order
• discrete distribution

Innovation

• Rényi entropy estimation
• sample complexity
• polynomial approximation
• moment matching
• noninteger order