APT: Adaptive Personalized Training for Diffusion Models with Limited Data

2025-07-03

AI Summary

To address three key challenges in few-shot personalized diffusion model training—overfitting, catastrophic forgetting of prior knowledge, and degradation of text-image alignment—this paper proposes the Adaptive Representation Regularization framework (APT). APT achieves holistic optimization through three innovations: (1) an overfitting-aware metric that dynamically modulates data augmentation intensity and loss weighting; (2) L2 regularization on the mean and variance of intermediate feature maps to stabilize representation distributions; and (3) explicit consistency constraints on cross-attention maps before and after fine-tuning to preserve semantic alignment. Under limited-data settings, APT effectively suppresses noise prediction drift, maintains denoising trajectory integrity, and consistently outperforms state-of-the-art methods in generation quality, diversity, and text fidelity.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Personalizing diffusion models using limited data presents significant challenges, including overfitting, loss of prior knowledge, and degradation of text alignment. Overfitting leads to shifts in the noise prediction distribution, disrupting the denoising trajectory and causing the model to lose semantic coherence. In this paper, we propose Adaptive Personalized Training (APT), a novel framework that mitigates overfitting by employing adaptive training strategies and regularizing the model's internal representations during fine-tuning. APT consists of three key components: (1) Adaptive Training Adjustment, which introduces an overfitting indicator to detect the degree of overfitting at each time step bin and applies adaptive data augmentation and adaptive loss weighting based on this indicator; (2)Representation Stabilization, which regularizes the mean and variance of intermediate feature maps to prevent excessive shifts in noise prediction; and (3) Attention Alignment for Prior Knowledge Preservation, which aligns the cross-attention maps of the fine-tuned model with those of the pretrained model to maintain prior knowledge and semantic coherence. Through extensive experiments, we demonstrate that APT effectively mitigates overfitting, preserves prior knowledge, and outperforms existing methods in generating high-quality, diverse images with limited reference data.

Problem

• Mitigates overfitting in diffusion models with limited data
• Preserves prior knowledge during model fine-tuning
• Maintains text alignment and semantic coherence

Innovation

• Adaptive Training Adjustment with overfitting indicator
• Representation Stabilization via feature map regularization
• Attention Alignment for prior knowledge preservation