Regularized Personalization of Text-to-Image Diffusion Models without Distributional Drift

2025-05-26

AI Summary

To address catastrophic forgetting—i.e., distribution shift induced by fine-tuning—in few-shot personalization of text-to-image diffusion models, this paper proposes a Lipschitz-constrained regularization framework. It is the first to explicitly model and suppress output distribution drift during diffusion model personalization, jointly optimizing a Lipschitz regularizer with CLIP/DINO feature-space alignment losses. This ensures faithful injection of novel subject concepts while preserving the pre-trained model’s generalization capability. Experiments under data-scarce conditions demonstrate that our method significantly outperforms existing approaches, achieving state-of-the-art performance across CLIP-T, CLIP-I, and DINO metrics. It effectively balances personalization fidelity and knowledge retention, offering verifiable stability guarantees for few-shot controllable image generation.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Privacy
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Personalization using text-to-image diffusion models involves adapting a pretrained model to novel subjects with only a few image examples. This task presents a fundamental challenge, as the model must not only learn the new subject effectively but also preserve its ability to generate diverse and coherent outputs across a wide range of prompts. In other words, successful personalization requires integrating new concepts without forgetting previously learned generative capabilities. Forgetting denotes unintended distributional drift, where the model's output distribution deviates from that of the original pretrained model. In this paper, we provide an analysis of this issue and identify a mismatch between standard training objectives and the goals of personalization. To address this, we propose a new training objective based on a Lipschitz-bounded formulation that explicitly constrains deviation from the pretrained distribution. Our method provides improved control over distributional drift and performs well even in data-scarce scenarios. Experimental results demonstrate that our approach consistently outperforms existing personalization methods, achieving higher CLIP-T, CLIP-I, and DINO scores.

Problem

• Adapt pretrained diffusion models to new subjects with few examples
• Prevent distributional drift while integrating new concepts
• Balance personalization and preservation of original generative capabilities

Innovation

• Lipschitz-bounded training objective prevents drift
• Regularized adaptation to novel subjects
• Improved performance in data-scarce scenarios