Semi-Supervised Learning with Noisy Proxy Covariates: Generalization Bounds and Distribution Regression

2026-05-30

AI Summary

This work addresses semi-supervised regression in settings where labeled data are scarce but abundant unlabeled data are available alongside noisy pre-trained representations serving as proxy covariates. The authors propose a two-stage estimator that first leverages all proxy covariates to learn kernel features and then fits a ridge regression predictor using the limited labeled samples. Under conditions of controllable proxy noise and sufficient unlabeled data, the method recovers the fast convergence rate dictated by the number of labeled samples. The theoretical analysis establishes finite-sample generalization bounds and unifies distribution regression within the same framework, providing corresponding performance guarantees. Empirical results demonstrate that the proposed approach significantly outperforms existing supervised and semi-supervised baselines, particularly in low-label regimes.

Abstract

In many modern machine learning pipelines, abundant pretrained representations serve as noisy proxy covariates, while task-specific labels remain scarce. We study semi-supervised regression in this setting, and propose a simple two stage estimator that learns kernel eigenfeatures from all proxy covariates and fits a ridge predictor on labeled data. We derive finite sample bounds showing that fast labeled sample rates are recovered when proxy perturbation is controlled and unlabeled proxy covariates are sufficiently abundant. We also show that distribution regression is a direct special case, with analogous guarantees when the finite bag size is large enough. Experiments show consistent gains over supervised and semi-supervised baselines, especially in low label regimes.

Problem

• Semi-Supervised Learning
• Noisy Proxy Covariates
• Distribution Regression
• Generalization Bounds

Innovation

• semi-supervised regression
• noisy proxy covariates
• kernel eigenfeatures
• generalization bounds
• distribution regression