Feature Selective Model Collapse in Diffusion Models: Total Replacement versus Fixed-Budget Training

2026-10-01

AI Summary

This study addresses the phenomenon of selective feature collapse in diffusion models under fixed-budget training, which contradicts conclusions drawn from full-replacement protocols. Specifically, it investigates the degradation mechanisms that arise when historical data is retained but the proportion of real samples is diluted. Through linear response modeling, stochastic recursion analysis, and benchmark experiments on datasets such as MNIST, the multi-generational parameter dynamics are rigorously examined. The findings reveal that under fixed-budget protocols, certain features remain robustly preserved over extended iterations, resulting only in selective loss rather than comprehensive degradation—a behavior markedly distinct from the rapid collapse observed under full-replacement settings. This discovery revises the universality assumption of conventional model collapse theory and offers a novel perspective for understanding feature evolution during the iterative training of generative models.

Abstract

Model collapse arises when generative models are trained on synthetic data produced by earlier models. The phenomenon has attracted considerable attention because of its societal and technical implications. However, previous studies have reached seemingly contradictory conclusions: replacing real data with synthetic data causes collapse (Shumailov et al.), yet accumulating real data alongside synthetic data can prevent it. For diffusion models, we study an intermediate regime typical of finite-budget pipelines: all past datasets and the real data are kept, but each new model is trained on a fixed-size sample from this growing pool, so the real fraction vanishes without any data being removed. Experiments on a 2D spiral dataset as well as the image benchmarks (MNIST, Fashion-MNIST, and CIFAR-10) show that replacement protocol degrades dataset rapidly as in the literature, whereas the fixed budget degrades only partially, sparing some features. A linear-response model of the multi-generational parameter dynamics, analyzed by stochastic recursion, confirms that the two protocols differ: some features will be fragile and lost within a few generations for both protocols, while some will be robust and preserved over practically unbounded horizons under the fixed budget protocol.

Problem

• Model Collapse
• Diffusion Models
• Synthetic Data
• Feature Selective Degradation
• Fixed-Budget Training

Innovation

• Model Collapse
• Diffusion Models
• Feature Selective Degradation
• Fixed-Budget Training
• Linear-Response Model