Asymptotic Analysis of Empirical Risk Minimization on Entry-wise i.i.d. Heavy-Tailed Data

2026-10-05

AI Summary

This study addresses the challenge of precisely characterizing the asymptotic prediction performance of empirical risk minimization under heavy-tailed data. By introducing functional order parameters to formulate a stochastic effective problem and integrating the replica method from statistical physics with high-dimensional asymptotic analysis, this work establishes heavy-tailed universality laws and scaling relations. In the high-dimensional limit, it provides a fully analytical characterization of both the generalization error and the Bayes-optimal prediction error for linear regression. These results overcome longstanding bottlenecks in the asymptotic analysis of heavy-tailed distributions and rigorously determine the scaling behavior governing prediction reliability. Ultimately, this research lays a solid theoretical foundation for understanding persistently locally heterogeneous systems.

Abstract

Many real-world datasets exhibit unusually large values far more frequently than predicted by Gaussian models. Heavy-tailed distributions capture this behavior, yet evaluating learning performance under them remains challenging because rare, large feature entries retain non-vanishing effects even in high dimensions. Even in the canonical setting of empirical risk minimization for linear regression with entry-wise i.i.d. symmetric $α$-stable data, a precise asymptotic characterization of prediction has been lacking. In this work, we introduce a functional order parameter that describes the random effective problem associated with each coefficient. Using the replica method, we fully characterize the generalization error in the proportional high-dimensional limit where the sample size and feature dimension diverge at a fixed ratio. Additionally, this analysis establishes a heavy-tail universality law, scaling laws relating typical errors to prediction reliability, and the Bayes-optimal prediction error. In addition to characterizing the effects of extreme entries on the learning process, our method applies broadly to other systems with persistent local heterogeneity.

Problem

• empirical risk minimization
• heavy-tailed data
• linear regression
• asymptotic analysis
• generalization error

Innovation

• Empirical Risk Minimization
• Heavy-Tailed Data
• Replica Method
• Functional Order Parameter
• High-Dimensional Asymptotics