A Characterization of List Regression

2024-09-28

arXiv.org

AI Summary

This work investigates the PAC sample complexity of list regression: given a class of real-valued functions, the learner outputs a list of $k$ predictions, and succeeds if at least one prediction is close to the true label. We establish the first tight characterizations for both the realizable and agnostic settings by introducing two novel combinatorial dimensions—the $k$-Optimal Index Growth (k-OIG) dimension and the $k$-fat-shattering dimension—each precisely capturing the minimal sample requirements. Both dimensions yield tight bounds and unify classical notions: the fat-shattering dimension for standard regression and the OIG dimension for list classification. Technically, our analysis integrates combinatorial dimension theory, generalization bound derivation for real-valued function classes, and a $k$-ary extension of the fat-shattering dimension. This work bridges a fundamental theoretical gap between list classification and regression, providing the first complete framework for the sample complexity of list regression.

Technology Category

• Machine Learning: Learning Theory
• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

There has been a recent interest in understanding and characterizing the sample complexity of list learning tasks, where the learning algorithm is allowed to make a short list of $k$ predictions, and we simply require one of the predictions to be correct. This includes recent works characterizing the PAC sample complexity of standard list classification and online list classification. Adding to this theme, in this work, we provide a complete characterization of list PAC regression. We propose two combinatorial dimensions, namely the $k$-OIG dimension and the $k$-fat-shattering dimension, and show that they characterize realizable and agnostic $k$-list regression respectively. These quantities generalize known dimensions for standard regression. Our work thus extends existing list learning characterizations from classification to regression.

Problem

• Characterizes sample complexity of list PAC regression.
• Introduces k-OIG and k-fat-shattering dimensions for regression.
• Extends list learning from classification to regression tasks.

Innovation

• Introduces $k$-OIG dimension for regression
• Introduces $k$-fat-shattering dimension for regression
• Extends list learning from classification to regression