Self-Consuming Generative Models with Co-Evolving Human Preferences

2026-10-07

AI Summary

This study addresses the instability and long-term bias amplification arising from the coupled co-evolution of user preferences and model distributions within the self-consuming loops of generative models. By modeling the iterative training process through dynamical systems theory, this work reveals that training exclusively on synthetic data tends to converge toward degenerate point equilibria. To mitigate this, a hybrid sampling strategy and joint optimization algorithm are proposed, which inject reference data and adaptively select mixing weights to guide system evolution. Theoretically, it is proven that incorporating reference data establishes a globally unique attractor, thereby enabling controllable dynamics. Furthermore, an efficient algorithm is designed to minimize data acquisition costs while steering the system toward stable equilibria that preserve desired properties.

Abstract

Generative models are increasingly trained in self-consuming iterative loops, where users curate preferred samples from model-generated candidates and the curated samples are used to train future generations of the model. Prior work has largely assumed fixed user preferences, but in practice exposure to model outputs gradually reshapes what users perceive as desirable, creating a feedback loop in which model distributions and user preferences co-evolve. We take a first step toward understanding the long-term behavior of such coupled dynamics. We show that when training relies entirely on user-curated synthetic data, iterative curation amplifies initial biases and drives the system toward one of multiple singleton equilibria in which the instance holding an initial advantage eventually dominates. In contrast, injecting reference data into training at a sufficiently large rate fundamentally changes the dynamics and yields a unique globally attracting equilibrium. Building on this insight, we study how reference-data injection can be used to control long-term outcomes, and propose an efficient algorithm that jointly selects a reference distribution and its mixing weight to steer the coupled system toward equilibria that preserve desired attributes while minimizing data collection costs.

Problem

• Generative Models
• Self-Consuming Loops
• Co-Evolving Preferences
• Coupled Dynamics
• Equilibrium

Innovation

• Generative Models
• Co-evolving Preferences
• Self-consuming Loops
• Reference-data Injection
• Equilibrium Dynamics