CLASP: Continual Low-rank Adapters for Spatially Placed Concepts from One Hypernetwork

2026-10-01

AI Summary

This study addresses the challenges of catastrophic forgetting and linear parameter growth in continual personalization of diffusion models. We propose a method that employs a fixed-size hypernetwork to dynamically generate low-rank adapters (LoRAs). This framework enables rehearsal-free continual learning and spatially controllable generation while keeping the base model frozen, with parameter overhead independent of the number of concepts and no additional components required. Experimental results demonstrate that our approach effectively overcomes limitations in long-sequence scalability, precisely preserves historical knowledge, and achieves accurate spatial localization. Overall, the proposed method outperforms or matches existing state-of-the-art techniques.

Abstract

Continual personalization of text-to-image diffusion models requires sequentially acquiring new concepts while retaining previously learned ones. However, existing methods either suffer from catastrophic forgetting or rely on storing additional concept-specific parameters and spatial components, causing their parameter footprint to grow with the concept stream. This limits their ability to scale to long sequences of personalization tasks. We propose a rehearsal-free approach that uses a single fixed-size hypernetwork to continually personalize a frozen diffusion model. Instead of expanding the model as new concepts are acquired, the hypernetwork dynamically produces the concept-specific adaptations required for personalization while preserving previously learned concepts. Our framework further integrates spatial control into the personalization process, allowing users to specify where a personalized concept should appear without introducing additional per-concept components. This formulation enables continual personalization with a parameter footprint that remains independent of the number of learned concepts, aside from compact concept representations. Experiments demonstrate strong retention of previously learned concepts and reliable spatial grounding, matching or improving upon existing methods while scaling effectively to long streams of personalization tasks.

Problem

• continual personalization
• text-to-image diffusion models
• catastrophic forgetting
• parameter scalability
• spatial control

Innovation

• Continual Personalization
• Hypernetwork
• Low-rank Adapters
• Spatial Control
• Text-to-Image Diffusion