Semi-Supervised Deep Sobolev Regression: Estimation and Variable Selection by ReQU Neural Network

2024-01-09

AI Summary

This paper addresses semi-supervised nonparametric regression, proposing the first provably optimal deep learning framework that simultaneously estimates both the regression function and its gradient. The method employs Rectified Quadratic Unit (ReQU) neural networks, incorporating Sobolev norm regularization and semi-supervised empirical risk minimization; unlabeled data are leveraged to approximate the gradient-norm regularization term, thereby enabling robust modeling under domain shift and facilitating nonparametric variable selection. Theoretically, the estimator achieves the minimax-optimal convergence rate in the L²-norm; the plug-in gradient estimator enjoys rigorous statistical guarantees; and a quantifiable semi-supervised gain is established. Numerical experiments demonstrate substantial performance improvements over baselines across diverse distribution shifts and high-dimensional sparse settings.

Technology Category

• Machine Learning: Semi-Supervised Learning
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

We propose SDORE, a Semi-supervised Deep Sobolev Regressor, for the nonparametric estimation of the underlying regression function and its gradient. SDORE employs deep ReQU neural networks to minimize the empirical risk with gradient norm regularization, allowing the approximation of the regularization term by unlabeled data. Our study includes a thorough analysis of the convergence rates of SDORE in $L^{2}$-norm, achieving the minimax optimality. Further, we establish a convergence rate for the associated plug-in gradient estimator, even in the presence of significant domain shift. These theoretical findings offer valuable insights for selecting regularization parameters and determining the size of the neural network, while showcasing the provable advantage of leveraging unlabeled data in semi-supervised learning. To the best of our knowledge, SDORE is the first provable neural network-based approach that simultaneously estimates the regression function and its gradient, with diverse applications such as nonparametric variable selection. The effectiveness of SDORE is validated through an extensive range of numerical simulations.

Problem

• Deep Learning
• Unsupervised Learning
• Feature Selection

Innovation

• SDORE
• ReQU Neural Networks
• Sobolev Regularization