Prediction-Only Distillation in Linear and Logistic Regression

2026-07-16

AI Summary

This work addresses self-distillation in a setting where only a pre-trained teacher model and unlabeled new data are available. The authors propose a prediction-only distillation framework that trains a student model using teacher-generated pseudo-labels and constructs a final predictor via an affine combination of the teacher’s and student’s predictions. Theoretically, they establish—for the first time under a prediction-only regime—that the risk of this hybrid predictor is strictly lower than that of the teacher alone, even in the presence of covariate distribution shift or anisotropy. They further show that the optimal mixing weights can be consistently estimated with only a small set of calibration labels. Methodologically, the approach integrates proportional asymptotic analysis, anisotropic ridge regression theory, and a logistic regression-based mixing strategy. Experiments demonstrate strictly improved risk in ridge regression and significant gains over both the teacher and standard distilled student models in classification tasks.

Abstract

Self-distillation (SD) is typically studied when the student is retrained on the teacher's original training inputs. In many practical deployments, however, the labeled training data are no longer available, and one has access only to the trained predictor and fresh unlabeled covariates. We study SD in this prediction-only regime through a fresh-X prediction-mixed scheme: a pure-distilled student is trained on fresh covariates pseudo-labeled by the teacher, and the final predictor is an affine combination of the teacher and student predictions. For ridge regression under proportional asymptotics, we derive deterministic equivalents for the optimally mixed prediction risk under general anisotropic covariance and deterministic signal. We show that this risk is strictly smaller than the teacher risk for almost every pair of teacher and student regularization levels, including when the fresh covariates are out-of-distribution and even when their covariance is isotropic. We further show that the optimal mixing weight cannot be identified from unlabeled data alone, but can be consistently estimated in a single post-training step using a small independent labeled calibration set, without additional model fitting. Finally, for binary logistic regression, we show that prediction mixing can outperform both the teacher and the pure-distilled classifier.

Problem

• self-distillation
• prediction-only
• unlabeled data
• prediction risk
• model calibration

Innovation

• prediction-only distillation
• self-distillation
• prediction mixing
• ridge regression
• logistic regression