covering number bounds

This work investigates the fundamental performance limits of learning and estimation tasks within an information-theoretic framework, independent of the computational capabilities of specific algorithms. By integrating tools from information theory and statistical learning theory—including metric entropy, VC dimension, Rademacher complexity, mutual information, and relative entropy—it systematically derives multiple upper bounds on generalization error. Simultaneously, leveraging Fano’s inequality together with covering and packing numbers, the study establishes information-theoretic lower bounds on minimax risk. The analysis unifies two complementary paradigms: one grounded in the geometric structure of metric spaces and the other based on information-theoretic measures. This synthesis yields a rigorous and broadly applicable theoretical framework for characterizing the optimal performance boundaries inherent to learning and estimation problems.

This work addresses the problem of efficiently estimating the partition function given only samples from a proposal distribution and access to the unnormalized density ratio of the target distribution. To this end, it introduces the novel concept of an “integral coverage profile,” which characterizes the concentration of the target distribution’s mass in regions of high density ratio. A general information-theoretic framework is established, relying solely on the f-divergence between the proposal and target distributions, and applicable to broad settings including heavy-tailed distributions. Under minimal assumptions, the framework rigorously distinguishes the complexity of approximate sampling from that of counting. By combining a generalized Paley–Zygmund inequality with importance sampling analysis, the paper derives tight upper and lower bounds on the sample complexity for multiplicative estimation of the partition function, unifying and extending classical results on importance sampling, rejection sampling, and heavy-tailed mean estimation, while providing sharper finite-sample guarantees.

This paper addresses the efficient compression and approximation of labeled data in metric spaces, without imposing additional structural assumptions. It introduces the *discrete-continuous modulus* as an intrinsic regularity measure to characterize the inherent smoothness of data approximation—its first application to this problem. Methodologically, it develops a sample-driven approximation framework, formulating data approximation as a stochastic sphere-covering and combinatorial optimization problem; it further constructs multilevel approximation spaces and an enhanced multilevel Monte Carlo algorithm for hierarchical, efficient approximation of statistically uncertain data. Theoretically, it establishes convergence guarantees for intrinsic data approximation. Algorithmically, it ensures computability of the discrete-continuous modulus and controllable computational complexity. Numerical experiments validate the method’s feasibility, stability, and theoretical convergence across diverse synthetic and real-world datasets.

This work addresses the sample complexity of Sample Average Approximation (SAA) for convex and strongly convex stochastic programming (SP) under standard SP assumptions—without requiring uniform Lipschitz continuity. Methodologically, it integrates convex analysis, stochastic optimization, and functional inequalities to bypass entropy-based arguments. The key contribution is the first tight, metric-entropy-free sample complexity bounds: $O(1/varepsilon^2)$ for convex SP and $O(1/varepsilon)$ for strongly convex SP—improving upon state-of-the-art bounds by a factor of $O(d)$ by eliminating dependence on covering numbers or Rademacher complexity. Theoretically, it reveals that SAA achieves nearly identical sample efficiency as stochastic mirror descent, bridging a long-standing gap in theoretical understanding. Numerical experiments validate the tightness of the bounds and demonstrate that SAA exhibits provably superior practical performance over stochastic mirror descent in non-Lipschitz settings.

This paper investigates entropy and mutual information estimation for streaming discrete data under finite memory constraints: given an i.i.d. sequence, how to estimate the distribution’s entropy using an $S$-state finite automaton with probability at least $1-delta$ and additive error at most $varepsilon$, and what is the tight characterization of the minimal $S$? Methodologically, it establishes the first asymptotically tight memory complexity bound for entropy estimation, proposes a randomized finite-state algorithm based on approximate counting and bias correction, and reduces the lower-bound analysis to uniformity testing. The theoretical contributions are: (i) a tight upper bound of $Oig(n(log n)^4/(varepsilon^2delta)ig)$ and a tight lower bound of $Omegaig(max{n,, log n / varepsilon}ig)$ on memory complexity for entropy estimation; and (ii) the first complete characterization of memory complexity for mutual information estimation, extending the results to that setting.

This work addresses the limitations of traditional Block Decomposition Method (BDM), which neglects inter-block dependencies, leading to redundant description lengths and an inability to capture shared structures. The authors propose an improved framework grounded in reusable program code, modeling inter-block dependencies as a descriptive resource optimization problem formalized through an “algorithmic attention” allocation mechanism. By integrating Coding Theorem Method (CTM), conditional algorithmic complexity, Shannon entropy, and combinatorial optimization, they construct a computable algorithmic attention model. Theoretically, the approach is shown to strictly outperform independent block descriptions in the presence of shared structures, with performance gains positively correlated to algorithmic mutual information, and the paper provides a feasible implementation pathway for practical application.

This work proposes and investigates the problem of entropy equivalence testing: given samples from two unknown distributions, determine whether their Shannon entropies are equal or differ by at least ε. The authors design a sample- and time-efficient algorithm that establishes the first optimal upper bound on sample complexity for this task. By integrating techniques from Shannon entropy estimation, hypothesis testing, and structural properties of Bayesian networks, the proposed method significantly outperforms conventional distribution closeness testing approaches. Furthermore, it yields the first non-trivial closeness testing algorithm for low-degree Bayesian networks, achieving substantial improvements in both sample and computational complexity.

This work addresses the lack of reliable statistical evaluation methods for generative models, which hinders the assessment of their generalization performance and the estimability of evaluation metrics from finite samples. The authors propose a theoretical framework that systematically analyzes the conditions under which common evaluation metrics are statistically estimable, distinguishing between test-class-based metrics and divergence-based metrics in finite-sample settings. Leveraging tools from integral probability metrics (IPMs), Rényi divergences, and fat-shattering dimension, they rigorously establish—for the first time—that IPMs induced by bounded test classes admit arbitrarily accurate estimation from finite samples, whereas KL and Rényi divergences, which depend on rare events, do not. This study provides a foundational theoretical basis and practical guidance for evaluating generative models.

This work proposes a unified measure of model complexity based on gradient similarity under input perturbations, applicable to both parametric and non-parametric models. Existing complexity metrics often struggle to balance theoretical rigor with computational efficiency; in contrast, the proposed measure achieves both while offering a coherent framework that subsumes classical notions such as polynomial degree and kernel lengthscale. The approach provides a novel interpretation of the double descent phenomenon by revealing how model complexity evolves throughout the training process. Through rigorous theoretical analysis and empirical validation across diverse model classes—including neural networks and random Fourier features—the method demonstrates strong mathematical grounding and practical scalability, effectively capturing the nuanced dynamics of complexity in modern machine learning systems.

This work addresses the problem of efficiently estimating the size of the union of multiple implicitly defined sets in the streaming model, encompassing classical settings such as discrete Klee measure, combinatorial test coverage estimation, and DNF model counting. The authors propose a sampling-based streaming algorithm that achieves, for the first time, an update time linear in the dimension $d$, resolving a long-standing open question in the field. Their approach reveals a fundamental trade-off between near-optimal space complexity and $P^{NP}$-level update time in coverage estimation. The algorithm uses $O(R \log|\Omega|)$ space and $O(R \log R \cdot \log(M/\delta) \cdot \log|\Omega|)$ update time, where $R = O(\varepsilon^{-2} \log(M/\delta))$, and has been formally verified in Lean 4.