Denoising Score Matching with Random Features: Insights on Diffusion Models from Precise Learning Curves

2025-02-01

AI Summary

This work investigates the generalization and memorization behaviors of denoising score matching (DSM) in diffusion models under high-dimensional, large-data, and multi-feature regimes. We analyze random feature networks approximating the score function under standard Gaussian data, and—crucially—derive exact analytical expressions for both training and test errors of DSM in the high-dimensional asymptotic limit. Leveraging tools from random matrix theory and statistical physics, we quantitatively characterize how model complexity $p$ and the number $m$ of noisy samples govern the generalization–memorization phase transition, precisely identifying the critical phase boundary. Our theoretical predictions exhibit excellent agreement with numerical experiments, uncovering the critical mechanism underlying the transition from generalization to memorization in the overparameterized regime. The results provide an interpretable theoretical foundation for diffusion model architecture design, training strategy optimization, and controllable generalization performance.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

We derive asymptotically precise expressions for test and train errors of denoising score matching (DSM) in generative diffusion models. The score function is parameterized by random features neural networks, with the target distribution being $d$-dimensional standard Gaussian. We operate in a regime where the dimension $d$, number of data samples $n$, and number of features $p$ tend to infinity while keeping the ratios $psi_n=frac{n}{d}$ and $psi_p=frac{p}{d}$ fixed. By characterizing the test and train errors, we identify regimes of generalization and memorization in diffusion models. Furthermore, our work sheds light on the conditions enhancing either generalization or memorization. Consistent with prior empirical observations, our findings indicate that the model complexity ($p$) and the number of noise samples per data sample ($m$) used during DSM significantly influence generalization and memorization behaviors.

Problem

• Generative Models
• Denoising Score Matching
• Generalization and Memorization

Innovation

• Random Feature Neural Networks
• De-noising Score Matching
• Generalization-Memorization Boundary