information-theoretic lower bounds

This work investigates the sample complexity of estimating Rényi entropy and min-entropy under high-dimensional discrete distributions. By establishing constructive upper bounds and information-theoretic lower bounds, it provides the first tight characterization of min-entropy estimation with sample complexity Θ(k log k), correcting prior erroneous claims. For integer-order Rényi entropy, matching upper and lower bounds are derived, revealing the critical role of the order α in determining sample complexity. The analysis leverages an unbiased falling-factorial estimator for α-wise collisions, a concentration inequality based on dyadic interval partitioning, and a hidden-heavy-coordinate construction. The results show that when 1.001 ≤ α ≤ c₀ log k, the sample complexity is Θ_{c₀}(αk^{1−1/α}), while for higher orders it becomes Θ_ε(k log k).

This paper addresses two long-standing open problems in distributed simple binary hypothesis testing: (1) whether interaction reduces sample complexity, and (2) whether existing sample complexity bounds under communication constraints are tight. To resolve them, we derive a tensorized lower bound on the Bayesian error, establish an inverse data-processing inequality for the Hellinger-λ divergence, and conduct reverse information-flow analysis. We rigorously prove—*for the first time*—that sequential interaction provides no reduction in the minimal sample requirement under a single-bit communication constraint per round. Consequently, we obtain the optimal, tight sample complexity: Θ(1/(ε² log(1/δ))), identical to the noninteractive setting. This result definitively refutes the “interaction gain” conjecture and establishes the fundamental limit of communication-constrained distributed hypothesis testing.

This work investigates the fundamental performance limits of learning and estimation tasks within an information-theoretic framework, independent of the computational capabilities of specific algorithms. By integrating tools from information theory and statistical learning theory—including metric entropy, VC dimension, Rademacher complexity, mutual information, and relative entropy—it systematically derives multiple upper bounds on generalization error. Simultaneously, leveraging Fano’s inequality together with covering and packing numbers, the study establishes information-theoretic lower bounds on minimax risk. The analysis unifies two complementary paradigms: one grounded in the geometric structure of metric spaces and the other based on information-theoretic measures. This synthesis yields a rigorous and broadly applicable theoretical framework for characterizing the optimal performance boundaries inherent to learning and estimation problems.

This work addresses the tightness of information-theoretic generalization error bounds with respect to sample size $n$, particularly the looseness of the individual-sample mutual information (ISMI) bound. To overcome the suboptimal $O(1/sqrt{n})$ convergence rate, we introduce, for the first time, an *excess risk assumption*, yielding a tight $O(1/n)$ fast-rate bound. Furthermore, we propose a novel generalization framework based on the $(eta,c)$-central condition, under which the mutual information term directly governs the convergence rate. We rigorously prove that this bound achieves the optimal $O(1/n)$ rate under standard assumptions. Empirical evaluation on canonical tasks—such as Gaussian mean estimation—demonstrates substantial improvements over existing information-theoretic bounds. The proposed framework thus bridges theoretical rigor with practical superiority, advancing both the tightness and applicability of information-theoretic generalization analysis.

This paper investigates entropy and mutual information estimation for streaming discrete data under finite memory constraints: given an i.i.d. sequence, how to estimate the distribution’s entropy using an $S$-state finite automaton with probability at least $1-delta$ and additive error at most $varepsilon$, and what is the tight characterization of the minimal $S$? Methodologically, it establishes the first asymptotically tight memory complexity bound for entropy estimation, proposes a randomized finite-state algorithm based on approximate counting and bias correction, and reduces the lower-bound analysis to uniformity testing. The theoretical contributions are: (i) a tight upper bound of $Oig(n(log n)^4/(varepsilon^2delta)ig)$ and a tight lower bound of $Omegaig(max{n,, log n / varepsilon}ig)$ on memory complexity for entropy estimation; and (ii) the first complete characterization of memory complexity for mutual information estimation, extending the results to that setting.

This study addresses the fundamental challenge of achieving optimal function approximation under limited evaluation data—a central problem in numerical analysis and machine learning. From the perspective of information-based complexity, the work systematically investigates function recovery under generalized sampling by integrating information-theoretic analysis, optimal recovery theory, and nonlinear, adaptive, and randomized sampling mechanisms. It uncovers intrinsic connections among diverse sampling strategies, characterizes the information-theoretic limits of function approximation given finite data, and proposes efficient algorithms and sampling schemes that approach these limits. The results provide foundational insights and a unified framework for optimal sampling theory.

This work establishes, for the first time, a rigorous equivalence between optimization under a B-bit quantized stochastic first-order oracle and interactive compressed Gaussian mean estimation, in the context of increasingly prevalent low-bit (e.g., FP8, MXFP4) pretraining. Leveraging Fisher information trace, the multivariate van Trees inequality, and sequential rate-distortion theory, the authors derive unconditional information-theoretic lower bounds: a communication complexity lower bound of \( T_B = \Omega(d) \) and a statistical complexity lower bound of \( T = \Omega(\sigma^2 d / \varepsilon^2) \), which together yield a tight product-form bound \( T = \Omega\big((\sigma^2 d / \varepsilon^2) \cdot \max\{1, d/B\}\big) \). These results precisely characterize the fundamental performance limits of low-bit optimization, refute prior conjectures regarding noise correlations, and nearly match the rates achieved by existing algorithms under bounded dynamic range.

This work proposes a unified framework for sampling from arbitrary log-concave distributions by integrating the In-and-Out algorithm with exponential boosting techniques. By refining the upper bound on the Poincaré constant of the boosted distribution, the method achieves near-optimal convergence rates—matching theoretical lower bounds—for two canonical settings: constrained distributions (e.g., Gaussian restricted to a convex body) and well-conditioned densities (e.g., strongly log-concave and smooth). Combining Poincaré inequality analysis with Markov chain Monte Carlo (MCMC) methodology, the approach efficiently samples from a warm-start initial distribution and establishes nearly tight, unified complexity upper bounds for log-concave sampling.

This work aims to derive tighter information-theoretic generalization error bounds to deepen the understanding of the generalization capability of randomized learning algorithms. By establishing a unified measure-change framework grounded in the data processing inequality for f-divergences, we propose a class of general and concise inequalities that flexibly adapt to diverse settings, including conditional mutual information, PAC-Bayes, and differential privacy. This approach not only simplifies and recovers several existing state-of-the-art results but also yields novel high-probability generalization bounds across multiple learning frameworks, significantly improving both the tightness and theoretical applicability of these bounds.

This study investigates a tight lower bound on the support size of input distributions that achieve the capacity of the binomial channel. By analyzing the structure of the output distribution and leveraging the asymptotic optimality of the Beta-binomial distribution, the authors establish a refined approximation linking channel capacity to the Beta-binomial law, employing tools from information-theoretic capacity analysis, relative entropy, and χ²-divergence comparisons. The main contribution is an improvement of the known lower bound on the support size from √n to the order of √(n log log n), proving that any capacity-achieving input distribution must contain at least this many mass points. Additionally, the paper provides an asymptotic expression for the channel capacity: C(n) = ½ log(nπ/2e) + o(1).