Data-driven Error Estimation: Upper Bounding Multiple Errors without Class Complexity as Input

2024-05-07

AI Summary

This paper addresses the problem of joint error control across multiple estimators. We propose a fully data-driven method that constructs simultaneous confidence intervals and bounds the maximum estimation error with high probability—without requiring prior knowledge of function class complexity (e.g., VC dimension or covering numbers). Our approach leverages empirical process theory and data resampling, integrating stochastic process control, concentration inequalities, and an error-aware exploration mechanism to adaptively capture unknown dependence structures among estimation errors. Unlike conventional union-bound or Talagrand-based methods, ours imposes no assumptions on pre-specified complexity measures and applies to both finite and infinite estimator classes. Experiments on multi-mean inference and contextual bandits demonstrate substantially tighter confidence intervals and over 30% reduction in conservatism, while rigorously preserving statistical validity.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Stochastic Optimization
• Intelligent Robots: State Estimation

Application Category

• Security and Privacy: Large-scale security measurements
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Constructing confidence intervals that are simultaneously valid across a class of estimates is central for tasks such as multiple mean estimation, bounding generalization error in machine learning, and adaptive experimental design. We frame this as an"error estimation problem,"where the goal is to determine a high-probability upper bound on the maximum error for a class of estimates. We propose an entirely data-driven approach that derives such bounds for both finite and infinite class settings, naturally adapting to a potentially unknown correlation structure of random errors. Notably, our method does not require class complexity as an input, overcoming a major limitation of existing approaches such as union bounding and bounds based on Talagrand's inequality. In this paper, we present our simple yet general solution and demonstrate its flexibility through applications ranging from constructing multiple simultaneously valid confidence intervals to optimizing exploration in contextual bandit algorithms.

Problem

• Estimates maximum error bounds without class complexity input.
• Adapts to unknown correlation structures in random errors.
• Applies to multiple confidence intervals and contextual bandits.

Innovation

• Data-driven error estimation without class complexity input
• Adapts to unknown correlation structures of random errors
• Applies to both finite and infinite class settings