On the entropy growth of sums of iid discrete random variables

2025-08-07

AI Summary

This paper investigates the asymptotic growth of the Shannon entropy of sums of $N$ i.i.d. discrete random variables, aiming to establish a universal lower bound in terms of $N$. To overcome reliance on the Central Limit Theorem, the authors introduce a novel distributional invariant—the *incommensurability rank* $r(X)$—defined for arbitrary discrete distributions. Their method integrates asymptotic entropy analysis for lattice-valued sums, multinomial approximation techniques, and advanced information-theoretic tools. They rigorously derive the asymptotic lower bound $H(S_N) ge frac{r(X)}{2}log N + C$, where the constant $C$ depends only on the marginal distribution of a single variable. When $r(X)=1$, the bound recovers classical results; for higher-rank distributions, it yields strictly tighter estimates. This work reveals, for the first time, an intrinsic connection between the entropy growth rate under addition and the algebraic structure of the underlying distribution, thereby providing a unifying theoretical framework for discrete entropy dynamics.

Technology Category

• Machine Learning: Information Theory
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Security and Privacy: Large-scale security measurements
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

We derive an asymptotic lower bound on the Shannon entropy $H$ of sums of $N$ arbitrary iid discrete random variables. The derived bound $H geq frac{r(X)}{2}log(N) + {it cst}$ is given in terms of the incommensurability rank $r(X)$ of the random variable -- a positive integer quantity that we introduce. The derivation does not rely on central limit theorems, but builds upon the known expressions of the asymptotic entropy of the multinomial distribution and sums of iid lattice random variables, which correspond to the case $r(X)=1$.

Problem

• Studies entropy growth in sums of iid discrete random variables
• Derives asymptotic lower bound for Shannon entropy H
• Introduces incommensurability rank r(X) as key metric

Innovation

• Derives entropy bound for iid sums
• Introduces incommensurability rank r(X)
• Uses multinomial distribution asymptotics