Non-asymptotic estimates of the minimal risk in statistical learning

2026-06-22

AI Summary

This study investigates high-confidence non-asymptotic upper and lower bounds for the minimal risk in statistical learning, circumventing the conventional reliance on boundedness assumptions of the empirical risk function. By integrating sharp forms of Talagrand’s concentration inequality—specifically the Bousquet and Klein–Rio refinements—with transport-entropy inequalities and empirical process theory, the authors derive a lower bound independent of both the number of parameters and input dimension, under Gaussian or exponential integrability assumptions. The corresponding upper bound is characterized by the interplay between sample size and the box-counting dimension of the parameter set measured in an Orlicz norm. This work thus provides a more general and non-asymptotic theoretical framework for evaluating learning algorithm performance without resorting to asymptotic approximations.

Abstract

In this paper we prove some concentration inequalities for two types of error probabilities in the Empirical Risk Principle (ERP) in statistical learning, which provide a lower bound and an upper bound for the minimal risk (in terms of the minimal empirical risk) with non-asymptotic high confidence. The usual boundedness condition of the empirical risk function is relaxed to the Gaussian or exponential integrability condition. The confidence of the lower bound of the minimal risk is shown to be independent of the number of training parameters and the dimension of the input vectors, allowing one to detect the deficiency of a learning machine efficiently; and the confidence of the upper bound of the minimal risk is proved to be high provided that the sample size $n$ is much greater than the box dimension of the parameter set $Θ$ in the Orlicz metric $d_{ψ_1}$ associated with the risk functions. Our work is based on Talagrand's concentration inequalities (the sharp versions by Bousquet and Klein-Rio), transport-entropy inequalities and the recent progress in the theory of empirical processes and statistical learning.

Problem

• non-asymptotic
• minimal risk
• empirical risk principle
• concentration inequalities
• statistical learning

Innovation

• non-asymptotic risk bounds
• empirical risk principle
• concentration inequalities
• Orlicz metric
• transport-entropy inequalities