HyperErase: Scale-Calibrated Hypernetwork for Multi-Concept Erasure in Text-to-Image Models

2026-09-25

AI Summary

This study addresses the limitations of existing concept erasure methods for text-to-image models, which rely on static weight paradigms that struggle with diverse prompts and suffer from parameter interference during multi-concept extension. To overcome these challenges, this work reformulates concept erasure as prompt-conditioned parameter amortization, proposing a hypernetwork-based dynamic parameter synthesis framework. The method optimizes LoRA adapter generation via a decoupled correction strategy and incorporates square-root scaling with teacher prior rectification to suppress overfitting. Crucially, it eliminates the need for gradient updates at inference, dynamically generating dedicated LoRA adapters for arbitrary prompts through a single forward pass. Experimental results demonstrate that the proposed approach achieves an excellent balance among erasure efficacy, image quality, and semantic alignment, matching state-of-the-art performance on single-concept benchmarks while effectively supporting efficient multi-concept erasure.

Abstract

Recent advances in text-to-image (T2I) generation have substantially improved visual synthesis, but have also raised increasing safety concerns due to their potential to generate harmful or undesirable content. Existing concept erasure methods predominantly follow a static weight paradigm, producing a single frozen adapter that struggles to adapt to diverse prompt variations and suffers from parameter interference when scaling to multiple concepts. We propose \textbf{HyperErase}, a framework for concept erasure based on hypernetwork-driven prompt-conditioned parameter synthesis. Our approach first reframes concept erasure as prompt-conditioned parameter amortization and trains a hypernetwork to map textual descriptions to prompt-specific LoRA updates, eliminating the need for per-prompt gradient optimization or manual LoRA merging. To further improve the stability and precision of synthesized adapters, we develop a decoupled rectification strategy, which disentangles LoRA tokens into pattern and scale subspaces, applies a square-root transform to curb multiplicative over-scaling, and leverages teacher-derived canonical priors for inference-time correction. Extensive experiments across major concept categories demonstrate that HyperErase consistently improves the trade-off between erasure effectiveness, image quality, and semantic alignment, achieving performance comparable to gold-standard single-concept baselines. Furthermore, the resulting models can provide specialized LoRAs for each input prompt variation in a single forward pass without requiring gradient updates during inference. These principled and flexible framework offers a new paradigm for concept erasure in T2I models.

Problem

• concept erasure
• text-to-image models
• multi-concept
• parameter interference
• safety

Innovation

• Hypernetwork
• Concept Erasure
• Prompt-Conditioned Parameter Synthesis
• Decoupled Rectification
• Text-to-Image Models