Attribute-Efficient PAC Learning of Sparse Halfspaces with Constant Malicious Noise Rate

2025-05-27

AI Summary

This work studies efficient PAC learning of sparse halfspaces—i.e., linear classifiers with an $s$-sparse weight vector $w^* in mathbb{R}^d$—under constant-rate malicious noise. To overcome the $O(d)$ sample complexity bottleneck of standard approaches, we propose a robust optimization algorithm based on hinge-loss minimization, integrating sparse regularization with a novel gradient analysis technique. Under mild distributional assumptions—namely, concentration and margin conditions—we achieve the first attribute-efficient PAC learner for this setting. Theoretically, our algorithm attains sample complexity $mathrm{poly}(s, log d)$, exponentially improving upon the classical $O(d)$ dependence on ambient dimension. It is computationally efficient (polynomial-time), provably robust to adversarial label noise, and breaks long-standing theoretical barriers for learning sparse halfspaces under malicious corruption.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Attribute-efficient learning of sparse halfspaces has been a fundamental problem in machine learning theory. In recent years, machine learning algorithms are faced with prevalent data corruptions or even adversarial attacks. It is of central interest to design efficient algorithms that are robust to noise corruptions. In this paper, we consider that there exists a constant amount of malicious noise in the data and the goal is to learn an underlying $s$-sparse halfspace $w^* in mathbb{R}^d$ with $ ext{poly}(s,log d)$ samples. Specifically, we follow a recent line of works and assume that the underlying distribution satisfies a certain concentration condition and a margin condition at the same time. Under such conditions, we show that attribute-efficiency can be achieved by simple variants to existing hinge loss minimization programs. Our key contribution includes: 1) an attribute-efficient PAC learning algorithm that works under constant malicious noise rate; 2) a new gradient analysis that carefully handles the sparsity constraint in hinge loss minimization.

Problem

• Learning sparse halfspaces with attribute efficiency
• Robustness to constant malicious noise rate
• Achieving poly(s, log d) sample complexity

Innovation

• Sparse halfspace learning with poly(s, log d) samples
• Hinge loss minimization under sparsity constraints
• Robust to constant malicious noise rate