On the Interpolation Effect of Score Smoothing

2025-02-26

AI Summary

This work investigates the origin of generalization in score-based diffusion models, specifically whether score function smoothing induces implicit interpolation rather than memorization. Method: Under a controlled setting where training data are uniformly distributed on a one-dimensional linear subspace, we analytically model score dynamics, solve the denoising ODE, and conduct experiments with L2- and spectral-regularized neural networks. Contribution/Results: We establish, for the first time, a causal relationship between score smoothing and intra-subspace interpolation: smoothed score functions guide denoising trajectories to continuously interpolate within the low-dimensional subspace spanned by training data, thereby avoiding overfitting. Theoretically, we prove that smoothing controls the denoising path geometry; empirically, we show that regularized networks learn score functions with significantly enhanced interpolation propensity and suppressed memorization. This work provides the first theory–experiment closed loop for understanding the generalization mechanism of diffusion models.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

Score-based diffusion models have achieved remarkable progress in various domains with the ability to generate new data samples that do not exist in the training set. In this work, we examine the hypothesis that their generalization ability arises from an interpolation effect caused by a smoothing of the empirical score function. Focusing on settings where the training set lies uniformly in a one-dimensional linear subspace, we study the interplay between score smoothing and the denoising dynamics with mathematically solvable models. In particular, we demonstrate how a smoothed score function can lead to the generation of samples that interpolate among the training data within their subspace while avoiding full memorization. We also present evidence that learning score functions with regularized neural networks can have a similar effect on the denoising dynamics as score smoothing.

Problem

• Examining interpolation effect in score-based diffusion models
• Studying score smoothing and denoising dynamics interplay
• Exploring regularized neural networks' impact on score functions

Innovation

• Score-based diffusion models
• Smoothing empirical score function
• Regularized neural networks