empirical risk minimization

This paper identifies an inherent suboptimality of Empirical Risk Minimization (ERM) under squared loss: its bias term dominates the estimation error, preventing attainment of the minimax optimal convergence rate; in contrast, the variance term achieves the minimax rate—a fact previously unverified rigorously. To address this, the authors establish, for the first time under random design, a non-asymptotic, sharp upper bound proving the minimax optimality of ERM’s variance term. They unify and extend Chatterjee’s admissibility theorem and the Caponnetto–Rakhlin stability result to realistic random-design settings. Furthermore, they systematically characterize the intrinsic irregularity of the empirical loss landscape for non-Donsker function classes. Integrating bias–variance decomposition, empirical process theory, and probabilistic analysis, the work delivers the most comprehensive and rigorous non-asymptotic risk decomposition for ERM to date, substantially advancing its theoretical foundations.

Traditional empirical risk minimization (ERM) suffers from poor generalization in over-parameterized regimes, struggling to balance fitting accuracy and robustness. Method: We propose functional risk minimization (FRM), a unified framework that defines risk over function space—not pointwise predictions—treating function approximation as the fundamental optimization unit. FRM constructs structured risk via parameterized function families $f_{ heta_i}$ and incorporates parsimony regularization, recovering ERM as a special case while more faithfully modeling noise processes. Contribution/Results: FRM provides a “minimal-function-fitting” interpretation of generalization for over-parameterized models. We prove its theoretical generality—encompassing major loss functions—and demonstrate empirically that FRM consistently outperforms ERM baselines across supervised, unsupervised, and reinforcement learning tasks, with particularly pronounced gains in over-parameterized settings.

This paper addresses uncertainty quantification for risk-minimizing estimators in machine learning, overcoming limitations of classical approaches that rely on restrictive distributional assumptions and asymptotic theory. We propose the first general-purpose, finite-sample, distribution-free, and frequentist-valid inference framework applicable to *any* risk minimizer. Our method is grounded in the generalized likelihood ratio test, integrated with empirical process analysis and data-driven tuning, and inherently supports anytime-valid inference. Theoretically, it guarantees exact coverage of confidence sets for *all* finite sample sizes—without asymptotic approximations. Empirically, it consistently outperforms classical asymptotic methods across diverse tasks, demonstrating both high accuracy and strong robustness. This work establishes a new paradigm for model-agnostic statistical inference.

This paper addresses the construction of confidence intervals for function-value learning in nonparametric estimation and stochastic programming—including SDE-based models. We systematically introduce the moderate deviations principle for the first time to develop high-precision confidence intervals that simultaneously achieve statistical optimality and computational tractability. The proposed method is theoretically optimal under multiple criteria: exponential accuracy, minimality, consistency, controlled misrepresentation probability, and uniform most accurate (UMA) inference. Crucially, the resulting confidence intervals admit a unified robust optimization formulation and—under broad model conditions—are exactly equivalent to finite-dimensional convex programs, thereby substantially improving both statistical efficiency and computational solvability. This work establishes a novel paradigm for nonparametric inference in complex stochastic systems.

This work addresses the family of parametric optimization problems and proposes the first unified, data-driven framework for analyzing the generalization performance of both classical and learned optimizers. Methodologically: (1) it introduces PAC-Bayes theory to the analysis of learned optimizers, deriving verifiable, high-probability generalization upper bounds; (2) it establishes performance bounds for classical optimizers based on empirical convergence rates; and (3) it pioneers a learning paradigm that directly minimizes the PAC-Bayes bound during training. Evaluated on signal processing, control, and meta-learning tasks, the derived bounds are significantly tighter than conventional worst-case guarantees. Moreover, the theoretical generalization guarantees for learned optimizers consistently exceed the empirical performance of their non-learned baselines—thereby unifying theoretical rigor with practical efficacy.

This work addresses the limitations of traditional generalization analyses, which rely on the often unverifiable assumption of independent and identically distributed (i.i.d.) data and thus struggle to accurately characterize model performance on unseen data. The paper proposes a deterministic generalization analysis framework that dispenses with any prior probabilistic assumptions. By examining the sensitivity of optimization solutions to data perturbations, it decomposes the generalization error into geometric and probabilistic components, achieving their first-ever decoupling. The framework expresses generalization bounds via a variational principle, leveraging deterministic perturbation analysis and optimization sensitivity theory to capture the discrepancy between in-sample and out-of-sample performance. Error terms are evaluated through posterior statistical hypotheses, enabling the recovery of conventional high-probability or expected generalization guarantees—all without requiring distributional assumptions.

This work addresses the lack of a unified framework for shrinkage, thresholding, and regularization methods in normal mean estimation by proposing a general class of estimators that encompasses both James–Stein-type and Lasso-type rules. Within this class, the authors derive the NOMAD estimator by minimizing a feasible, data-driven approximation to the risk. The approach is extended to settings with correlated observations and linear regression. A key innovation is the establishment of a unified risk minimization framework featuring a data-adaptive penalty, which—remarkably—achieves approximate risk consistency in correlated normal mean models for the first time. Theoretical analysis confirms that the estimator is consistent under both independent and correlated designs, admits an equivalent penalized regression formulation, and either subsumes or improves upon several classical methods in theory.

This work addresses the challenge of accurately estimating the score function of a probability measure from limited samples to enhance the generation quality of score-based generative models while mitigating overfitting. To this end, it introduces Sobolev space regularization into score function estimation for the first time, establishing a learning framework based on empirical risk minimization. The proposed method achieves minimax-optimal estimation rates on the flat torus and provides theoretical guarantees for score-based generative models, effectively balancing estimation accuracy with generalization capability.

Although high-dimensional non-convex empirical risk functions possess numerous local minima, gradient-based algorithms often converge to solutions near the global optimum; however, the precise characterization of the polynomial-time reachable region remains elusive. This work addresses this gap in the context of multi-index supervised learning models by integrating replica symmetry breaking theory with the Incremental Approximate Message Passing (IAMP) algorithm. Through high-dimensional asymptotic analysis within the empirical risk minimization framework, the study precisely characterizes the training error achievable by IAMP and establishes its quantitative relationship with the test error. The results delineate the performance boundary between computational feasibility and statistical optimality, demonstrating that IAMP achieves optimal performance among all polynomial-time algorithms.

This study addresses the optimization of optimized certainty equivalents (OCE), a risk measure widely employed in portfolio optimization and uncertainty quantification in machine learning. Focusing on unbounded random variables, it establishes the first OCE optimization framework by leveraging the duality between OCE and utility-based shortfall risk (UBSR). The work introduces a sample average approximation (SAA)-based OCE estimator and a corresponding stochastic gradient algorithm. Theoretical contributions include the design of an OCE gradient estimator, non-asymptotic mean squared error bounds, and convergence rates for the proposed algorithm. Empirical evaluations on portfolio optimization and uncertainty quantification tasks demonstrate the method’s effectiveness, offering both strong theoretical guarantees and practical performance.