Rectify Then Diffuse: Disentangling Concepts Before Denoising Trajectory Unfolds

2026-08-04

AI Summary

This work addresses the challenge in multi-concept text-to-image generation where early-stage denoising often suffers from attention coupling due to overlapping concept representations in latent space, leading to concept omission or erroneous fusion. To resolve this, the authors propose Rectify-then-Diffuse, a training-free framework that formulates compositional generation as a boundary condition problem. It applies a one-time correction to the initial latent variables prior to standard denoising to achieve concept disentanglement. The approach integrates a Soft-Overlap Disentanglement objective with an Isotropic Gradient Rectification strategy, enabling unified concept assignment while preserving scale consistency in latent-space displacements. Evaluated on the AE-Bench object-pair subset, the method improves BLIP-VQA scores by 45.8% and ImageReward by 19.6%, while achieving a 2.3× faster inference speed compared to CO3.

Abstract

Text-to-image diffusion models can generate individual concepts well, but they often omit or merge concepts incorrectly with multiple concepts. We trace these failures to an early coordination bottleneck: before denoising begins, prompt-conditioned attention may allocate different concepts to strongly overlapping spatial support, which can keep their attention coupled as denoising proceeds. This observation motivates treating compositional generation as a boundary-condition problem rather than repeatedly controlling the evolving trajectory. To this end, we propose Rectify-then-Diffuse (RTD), a training-free framework that rectifies the initial allocation once before standard denoising. Firstly, we propose Soft-Overlap Disentanglement (SOD), which converts normalized overlap between pilot concept maps into a differentiable and layout-agnostic separation objective. Secondly, we introduce Isotropic Gradient Rectification (IGR), which normalizes the SOD gradient and applies a bounded latent displacement with a consistent scale across prompts and initializations. Extensive experiments show that RTD achieves state-of-the-art compositional fidelity and robust gains. On the AE-Bench object pair subset, RTD improves BLIP-VQA by 45.8% and ImageReward by 19.6% over CO3 while running 2.3$\times$ faster. Code will be released at https://github.com/Z-yiwei/rectify-then-diffuse

Problem

• text-to-image diffusion
• compositional generation
• concept disentanglement
• attention coupling
• spatial overlap

Innovation

• Rectify-then-Diffuse
• Soft-Overlap Disentanglement
• Isotropic Gradient Rectification
• compositional generation
• diffusion models