Crafting Your Evolving Dreams: Concept-Incremental Versatile Customization

2026-06-03

AI Summary

Existing personalized diffusion models often suffer from catastrophic forgetting and concept neglect when continuously learning new user-specific concepts. To address this, this work proposes the Continually Customizable Diffusion Model (CCDM), which mitigates forgetting through Attribute-Decoupled LoRA (AD-LoRA) and a relevance-guided LoRA aggregation mechanism. CCDM further introduces a controllable region-aware context composition strategy to enable semantically independent yet smoothly blended integration of multiple concepts within designated image regions. As the first customizable diffusion framework supporting incremental concept expansion, CCDM efficiently learns new concepts while preserving previously acquired ones. Experimental results demonstrate that CCDM significantly outperforms current baselines, achieving superior consistency and controllability in multi-concept generation under continual customization settings.

Abstract

Custom diffusion models (CDMs) have garnered significant interest owing to their remarkable capacity for generating personalized concepts. However, the majority of CDMs unrealistically presume that the user's collection of personalized concepts is static and incapable of incremental growth over time. Furthermore, they exhibit significant catastrophic forgetting and concept neglect of previously learned concepts when incrementally learning a sequence of new ones. To resolve the above challenges, we develop a novel Continually Customizable Diffusion Model (CCDM), enabling users to perform concept-incremental versatile customization. Specifically, we design an attribute-decoupled LoRA (AD-LoRA) module and a relevance-guided AD-LoRA aggregation strategy to mitigate catastrophic forgetting. They can preserve concept-specific attributes of each task and leverage beneficial inter-task correlations to enhance the continual learning of new customization tasks. Additionally, to address the challenge of concept neglect, we propose a controllable regional context synthesis strategy that performs multi-concept composition in alignment with user-provided conditions. This strategy enhances the overall consistency in multi-concept synthesis by guaranteeing semantic independence between user-defined regions and their smooth boundary transitions. Experiments show our CCDM exhibits significant improvements over baseline methods.

Problem

• concept-incremental learning
• catastrophic forgetting
• concept neglect
• custom diffusion models
• continual learning

Innovation

• Continual Learning
• Custom Diffusion Models
• Catastrophic Forgetting
• Concept-Incremental Customization
• Attribute-Decoupled LoRA