empirical process theory

This work addresses the construction of confidence intervals for the mean of a finite or general-alphabet population under sampling without replacement. Leveraging the large deviation rate function for sampling without replacement, the authors propose a novel method that constructs confidence intervals via an empirical inverse rate function and derive its dual formulation to enable efficient computation. This approach uniquely links the interval width directly to the inverse of the rate function and achieves theoretical lower bounds across various asymptotic regimes. In the finite-alphabet setting, the resulting intervals attain the optimal width up to constant factors; for populations supported on $[0,1]$ or in smooth Banach spaces, the method further yields confidence intervals that are almost surely or non-asymptotically optimal.

This work establishes high-probability regret bounds for empirical risk minimization (ERM) and extends them to learning problems involving nuisance components, such as causal inference, missing data, and domain adaptation. By employing a three-step approach—elementary inequality, localized uniform concentration bounds, and a fixed-point argument—combined with a key radius defined via local Rademacher complexity, the study characterizes convergence rates in a modular analytical framework. This framework unifies the treatment of standard and nuisance-augmented ERM, explicitly decomposing statistical and approximation errors, and provides sufficient conditions for fast convergence. It recovers classical rates for VC-subgraph classes, Sobolev/Hölder spaces, and bounded variation function classes, and delivers transferable regret guarantees for orthogonal learning settings.

High-complexity machine learning models lack reliable, theoretically grounded mechanisms for detecting overfitting. Method: We propose a statistical hypothesis test that operates solely on training data, dispensing with the need for an independent validation set or PAC-style uniform convergence assumptions. Our approach formalizes overfitting via empirical mean consistency and constructs a rigorous testing framework based on Hoeffding-type concentration inequalities. Contribution/Results: This is the first method to use empirical mean consistency as an overfitting criterion, enabling significance-based inference and implicit diagnosis sensitive to distributional shifts. We prove its validity under mild regularity conditions. Empirical evaluation demonstrates robust identification of overfitting transition points and latent distribution drift, substantially improving both the reliability and interpretability of model selection.

This paper addresses the construction of confidence intervals for function-value learning in nonparametric estimation and stochastic programming—including SDE-based models. We systematically introduce the moderate deviations principle for the first time to develop high-precision confidence intervals that simultaneously achieve statistical optimality and computational tractability. The proposed method is theoretically optimal under multiple criteria: exponential accuracy, minimality, consistency, controlled misrepresentation probability, and uniform most accurate (UMA) inference. Crucially, the resulting confidence intervals admit a unified robust optimization formulation and—under broad model conditions—are exactly equivalent to finite-dimensional convex programs, thereby substantially improving both statistical efficiency and computational solvability. This work establishes a novel paradigm for nonparametric inference in complex stochastic systems.

This paper identifies an inherent suboptimality of Empirical Risk Minimization (ERM) under squared loss: its bias term dominates the estimation error, preventing attainment of the minimax optimal convergence rate; in contrast, the variance term achieves the minimax rate—a fact previously unverified rigorously. To address this, the authors establish, for the first time under random design, a non-asymptotic, sharp upper bound proving the minimax optimality of ERM’s variance term. They unify and extend Chatterjee’s admissibility theorem and the Caponnetto–Rakhlin stability result to realistic random-design settings. Furthermore, they systematically characterize the intrinsic irregularity of the empirical loss landscape for non-Donsker function classes. Integrating bias–variance decomposition, empirical process theory, and probabilistic analysis, the work delivers the most comprehensive and rigorous non-asymptotic risk decomposition for ERM to date, substantially advancing its theoretical foundations.

This work addresses the limitations of traditional generalization analyses, which rely on the often unverifiable assumption of independent and identically distributed (i.i.d.) data and thus struggle to accurately characterize model performance on unseen data. The paper proposes a deterministic generalization analysis framework that dispenses with any prior probabilistic assumptions. By examining the sensitivity of optimization solutions to data perturbations, it decomposes the generalization error into geometric and probabilistic components, achieving their first-ever decoupling. The framework expresses generalization bounds via a variational principle, leveraging deterministic perturbation analysis and optimization sensitivity theory to capture the discrepancy between in-sample and out-of-sample performance. Error terms are evaluated through posterior statistical hypotheses, enabling the recovery of conventional high-probability or expected generalization guarantees—all without requiring distributional assumptions.

This study investigates the robust stability of statistical estimators under η-fraction data corruption. To this end, it introduces “empirical sensitivity” as a novel metric quantifying an estimator’s susceptibility to data perturbations, with a focus on Gaussian mean estimation. Theoretical analysis establishes, for the first time, a lower bound of Ω(η + √(ηd/n)) on empirical sensitivity and demonstrates its tightness up to logarithmic factors. By integrating the Efron–Stein inequality with recent advances in robust mean estimation algorithms, the authors construct an estimator achieving a matching upper bound, thereby confirming the attainability of this limit. This work provides a new theoretical framework and quantitative tool for analyzing the stability of robust estimators.

This work investigates the minimax sample complexity of multi-calibration in the batch setting—specifically, the minimum number of samples required to achieve expected calibration error (ECE) at most ε uniformly over a given class of groups. By constructing randomized predictors via online-to-batch conversion and employing minimax analysis, the authors develop a unified framework applicable to weighted Lₚ multi-calibration (1 ≤ p ≤ 2) and elicitable properties such as expectiles and bounded-density quantiles. Their main contributions include establishing a Θ̃(ε⁻³) sample complexity when |G| ≤ ε⁻ᵏ for any κ > 0—significantly higher than the Θ̃(ε⁻²) rate for marginal calibration—and revealing a sharp threshold at κ = 0 where the complexity transitions from ε⁻² to ε⁻³. They also provide tight upper and lower bounds with exponent 3/p for Lₚ metrics, precisely characterizing the complexity in both batch and online settings.

This study addresses the challenge of constructing theoretically guaranteed prediction intervals for stationary Markov processes without relying on specific model structures. The authors propose Markov Distributional Conformal Prediction (MDCP), a method that estimates the transition distribution function and applies a probability integral transform to convert the Markov sequence into approximately independent and identically distributed samples, thereby embedding it within a model-free distributional conformal prediction framework. This work is the first to extend distributional conformal prediction to Markov processes, establishing non-asymptotic unconditional coverage error bounds and proving asymptotic validity of conditional prediction intervals under a mild \(L^p\)-m-approximability condition. Empirical results demonstrate that MDCP significantly outperforms baseline approaches such as model-free bootstrap methods in finite-sample settings.

Although high-dimensional non-convex empirical risk functions possess numerous local minima, gradient-based algorithms often converge to solutions near the global optimum; however, the precise characterization of the polynomial-time reachable region remains elusive. This work addresses this gap in the context of multi-index supervised learning models by integrating replica symmetry breaking theory with the Incremental Approximate Message Passing (IAMP) algorithm. Through high-dimensional asymptotic analysis within the empirical risk minimization framework, the study precisely characterizes the training error achievable by IAMP and establishes its quantitative relationship with the test error. The results delineate the performance boundary between computational feasibility and statistical optimality, demonstrating that IAMP achieves optimal performance among all polynomial-time algorithms.