CountDiffusion: Text-to-Image Synthesis with Training-Free Counting-Guidance Diffusion

2025-05-07

AI Summary

To address the prevalent issue of inaccurate object counting in text-to-image (T2I) generation, this paper proposes a training-free, two-stage counting-guided diffusion framework. First, a pre-trained visual counting model (e.g., CountViT) performs real-time object counting on intermediate denoised images. Second, a gradient-guided correction module—built upon self-attention maps—dynamically refines the diffusion process to precisely align with the target count. This work introduces the first *training-free* counting guidance mechanism: it is plug-and-play, requiring no fine-tuning or retraining of existing T2I diffusion models (e.g., Stable Diffusion). Notably, it establishes the first general-purpose synergy between attention-based guidance and external counting modules. Evaluated across multiple benchmarks, the method improves object counting accuracy by 32.7% relatively, while preserving image fidelity and full compatibility with off-the-shelf diffusion models.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Stable Diffusion has advanced text-to-image synthesis, but training models to generate images with accurate object quantity is still difficult due to the high computational cost and the challenge of teaching models the abstract concept of quantity. In this paper, we propose CountDiffusion, a training-free framework aiming at generating images with correct object quantity from textual descriptions. CountDiffusion consists of two stages. In the first stage, an intermediate denoising result is generated by the diffusion model to predict the final synthesized image with one-step denoising, and a counting model is used to count the number of objects in this image. In the second stage, a correction module is used to correct the object quantity by changing the attention map of the object with universal guidance. The proposed CountDiffusion can be plugged into any diffusion-based text-to-image (T2I) generation models without further training. Experiment results demonstrate the superiority of our proposed CountDiffusion, which improves the accurate object quantity generation ability of T2I models by a large margin.

Problem

• Generating images with accurate object counts from text
• Overcoming training challenges in quantity-aware image synthesis
• Enhancing diffusion models without additional training for counting

Innovation

• Training-free counting-guidance diffusion framework
• Two-stage correction with counting model
• Plug-in for any diffusion-based T2I models