🤖 AI Summary
This work addresses the dual challenges of precise attribute control and cross-iteration consistency in multi-instance image generation. To this end, we propose (i) the MIGC++ controller, enabling text- or image-driven attribute control and bounding-box- or mask-driven spatial localization; and (ii) the Consistent-MIG algorithm, which preserves both unedited regions and instance identity across editing iterations. Our approach employs a divide-and-conquer single-instance diffusion architecture, cross-modal conditional injection, and iterative consistency regularization. We introduce two new benchmarks—COO-MIG and Multimodal-MIG—to rigorously evaluate multi-instance controllability. Extensive experiments on COCO-MIG, Multimodal-MIG, COCO-Position, and DrawBench demonstrate state-of-the-art performance, with significant improvements in positional accuracy, attribute fidelity, and the number of controllable instances. Notably, our method achieves, for the first time, unified fine-grained attribute isolation and stable iterative editing.
📝 Abstract
We introduce the Multi-Instance Generation (MIG) task, which focuses on generating multiple instances within a single image, each accurately placed at predefined positions with attributes such as category, color, and shape, strictly following user specifications. MIG faces three main challenges: avoiding attribute leakage between instances, supporting diverse instance descriptions, and maintaining consistency in iterative generation. To address attribute leakage, we propose the Multi-Instance Generation Controller (MIGC). MIGC generates multiple instances through a divide-and-conquer strategy, breaking down multi-instance shading into single-instance tasks with singular attributes, later integrated. To provide more types of instance descriptions, we developed MIGC++. MIGC++ allows attribute control through text & images and position control through boxes & masks. Lastly, we introduced the Consistent-MIG algorithm to enhance the iterative MIG ability of MIGC and MIGC++. This algorithm ensures consistency in unmodified regions during the addition, deletion, or modification of instances, and preserves the identity of instances when their attributes are changed. We introduce the COCO-MIG and Multimodal-MIG benchmarks to evaluate these methods. Extensive experiments on these benchmarks, along with the COCO-Position benchmark and DrawBench, demonstrate that our methods substantially outperform existing techniques, maintaining precise control over aspects including position, attribute, and quantity.