🤖 AI Summary
In high-dimensional linear regression, where the number of parameters is comparable to the sample size, model-averaged predictions are susceptible to the double descent phenomenon and local risk divergence. This work reveals an emergent smoothing effect induced by weighted model aggregation near the interpolation threshold, which effectively mitigates risk divergence. Building on this insight, the authors propose Large Model Averaging (LaMA), a method that optimally balances bias and asymptotic variance to enhance both fitting accuracy and generalization performance. Theoretical analysis leverages random matrix theory and high-dimensional asymptotic risk characterizations, while empirical evaluations on both synthetic and real-world datasets demonstrate that LaMA significantly improves predictive accuracy, confirming its superiority over existing approaches.
📝 Abstract
This paper investigates the predictive performance of model averaging in high-dimensional linear regression where the number of regressors is comparable to the sample size. We demonstrate that the double descent trajectory manifests within the model averaging framework, where the ensemble inherits the variance explosion of individual models near the interpolation boundary. However, we reveal that weighted aggregation simultaneously triggers an emergent smoothing effect that structurally suppresses the localized risk divergence, indicating that strategic weight choice serves as a vital stabilizing mechanism. Leveraging tools from random matrix theory, we derive the exact limiting out-of-sample risk under a nested model setting and provide a comprehensive characterization of the risk landscape. Building on these asymptotic results, we propose the Large Model Averaging (LaMA) method, which introduces a novel criterion incorporating in-sample bias and asymptotic out-of-sample variance to balance fitting accuracy and generalization. Numerical studies and real data applications confirm that LaMA achieves superior predictive accuracy in high-dimensional environments.