InstancePin: Instance-Addressable Layout-to-Image Diffusion via Coordinate Pinning

2026-08-01

AI Summary

Existing layout-to-image generation methods struggle to disentangle multiple instances of the same semantic class in dense urban scenes, often resulting in ambiguous boundaries and feature confusion. To address this, this work proposes an instance-addressable diffusion generation framework that enables fine-grained, instance-level spatial control without modifying the pretrained backbone. The approach introduces explicit coordinate anchors for each object, incorporating Fourier-encoded coordinate tokens, a coordinate-pinning attention mechanism, an instance-mask-guided feature fusion module, and dedicated instance-aware adapters. Evaluated on Cityscapes, the method substantially mitigates instance entanglement while enhancing both photorealism and semantic consistency of the generated images.

Abstract

Layout-to-image diffusion models have achieved impressive semantic controllability by conditioning generation on category-level segmentation maps. However, such category-aligned control is not necessarily instance-addressable: multiple nearby objects from the same category are often treated as a shared semantic region, leading to ambiguous boundaries, averaged appearances, and feature confusion among instances. This limitation is particularly evident in urban scene synthesis, where small and crowded pedestrians or vehicles require fine-grained instance separation while preserving global scene consistency. In this paper, we propose InstancePin, an instance-addressable layout-to-image diffusion framework that pins each object instance with an explicit coordinate anchor. Instead of directly injecting instance masks into the pretrained backbone, InstancePin introduces an independent instance-aware adapter to preserve the category-level generation prior while learning instance-specific spatial control. For each instance, its center coordinate is encoded with Fourier features and projected into a coordinate token, which serves as a spatial anchor queried by latent image features through coordinate pinning attention. To make these anchors spatially meaningful, we further supervise the coordinate attention maps with instance regions, encouraging each coordinate token to activate its corresponding object area. Finally, an instance-mask guided fusion module routes pretrained backbone features to non-instance regions and adapter features to instance regions, enabling local instance refinement without sacrificing global semantic fidelity. Extensive experiments on Cityscapes demonstrate that InstancePin mitigates instance entanglement in dense layouts and improves both image fidelity and semantic consistency.

Problem

• instance-addressable generation
• layout-to-image diffusion
• instance entanglement
• semantic controllability
• urban scene synthesis

Innovation

• instance-addressable generation
• coordinate pinning
• layout-to-image diffusion
• instance-aware adapter
• spatial anchor