Concentration Inequalities for Suprema of Empirical Processes with Dependent Data via Generic Chaining with Applications to Statistical Learning

2025-11-01

AI Summary

This paper addresses the challenge of establishing concentration bounds for suprema of empirical processes under dependent data—such as time series—in high-dimensional, heavy-tailed (sub-Weibull) settings. We propose the first unified analytical framework integrating generic chaining with coupling techniques, relaxing both the i.i.d. assumption and light-tailed restrictions. Our method yields sharp, non-asymptotic upper bounds on empirical process suprema. We apply these bounds to nonlinear regression and single-layer neural networks, demonstrating that empirical risk minimization achieves the optimal prediction error rate—identical to the i.i.d. case—even under temporal dependence. The core contribution is a novel paradigm for concentration inequalities tailored to dependent, high-dimensional, heavy-tailed data, providing a rigorous non-asymptotic foundation for statistical learning with dependent observations.

Technology Category

• Machine Learning: Other Foundations of Machine Learning
• Natural Language Processing: Learning & Optimization for NLP
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

This paper develops a general concentration inequality for the suprema of empirical processes with dependent data. The concentration inequality is obtained by combining generic chaining with a coupling-based strategy. Our framework accommodates high-dimensional and heavy-tailed (sub-Weibull) data. We demonstrate the usefulness of our result by deriving non-asymptotic predictive performance guarantees for empirical risk minimization in regression problems with dependent data. In particular, we establish an oracle inequality for a broad class of nonlinear regression models and, as a special case, a single-layer neural network model. Our results show that empirical risk minimzaton with dependent data attains a prediction accuracy comparable to that in the i.i.d. setting for a wide range of nonlinear regression models.

Problem

• Develops concentration bounds for dependent empirical process suprema
• Establishes non-asymptotic guarantees for dependent data regression
• Shows ERM achieves comparable accuracy to i.i.d. settings

Innovation

• Generic chaining with coupling for dependent data
• High-dimensional heavy-tailed sub-Weibull data framework
• Nonlinear regression neural network prediction guarantees