The Zero Pattern of a Design Matrix Drives Multiple Descent in Over-parameterized Regression

2026-07-27

AI Summary

This work addresses the limitations of classical overparameterized linear regression theory in explaining the multiple descent phenomenon observed in real-world data, which arises from covariate dependencies and covariance degeneracy. The authors introduce a novel framework that models the zero pattern of the design matrix as a bipartite graph and, for the first time, leverages the Dulmage–Mendelsohn decomposition and maximum matching theory. Combining random matrix theory with deterministic equivalent analysis in the zero-regularization limit, they rigorously derive an explicit expression for the prediction risk. This approach precisely characterizes how singular structures in the covariance matrix induce multiple peaks in the risk curve and accurately predicts both the locations and conditions under which multiple descents occur.

Abstract

Over-parameterized linear regression has been widely studied over the last decade. However, most existing works assume that the covariates are independent and that their covariance matrices are non-degenerate. In this paper, we relax both assumptions and derive deterministic equivalents for the prediction risk in a vanishing-ridge regime. We show that degeneracy of the covariance matrices and dependence can lead to multiple descent, and characterize where the corresponding peaks can occur. Our proofs use a novel graph representation of the variance profile. We show that maximum matchings and the Dulmage--Mendelsohn decomposition of the associated bipartite graph identify the configurations at which the variance becomes singular.

Problem

• over-parameterized regression
• covariance degeneracy
• multiple descent
• dependent covariates
• prediction risk

Innovation

• over-parameterized regression
• multiple descent
• graph representation
• Dulmage–Mendelsohn decomposition
• variance singularity