Technology Category
Computer Vision: Diffusion Models for VisionMachine Learning: Large Multimodal Models (LMMs)Natural Language Processing: (Large) Language Models
Explore In-Context Segmentation via Latent Diffusion Models
🤖 AI Summary
This work addresses zero-shot in-context segmentation by proposing the first latent diffusion model (LDM)-based framework for the task. Methodologically: (1) it introduces an instruction-driven cross-modal alignment mechanism to map reference images to target segmentation masks semantically; (2) it adopts a two-stage mask strategy to prevent information leakage during inference; and (3) it formulates an enhanced pseudo-mask supervision objective that jointly optimizes generation fidelity and segmentation accuracy. Contributions include: (i) the first extension of LDMs to in-context segmentation; (ii) the first fair, unified benchmark covering both image and video segmentation scenarios; and (iii) state-of-the-art performance on this benchmark—outperforming specialized segmentation models and mainstream vision foundation models—thereby demonstrating the feasibility of unifying segmentation and generative modeling within a single diffusion-based architecture.
Technology Category
Application Category
📝 Abstract
In-context segmentation has drawn increasing attention with the advent of vision foundation models. Its goal is to segment objects using given reference images. Most existing approaches adopt metric learning or masked image modeling to build the correlation between visual prompts and input image queries. This work approaches the problem from a fresh perspective - unlocking the capability of the latent diffusion model (LDM) for in-context segmentation and investigating different design choices. Specifically, we examine the problem from three angles: instruction extraction, output alignment, and meta-architectures. We design a two-stage masking strategy to prevent interfering information from leaking into the instructions. In addition, we propose an augmented pseudo-masking target to ensure the model predicts without forgetting the original images. Moreover, we build a new and fair in-context segmentation benchmark that covers both image and video datasets. Experiments validate the effectiveness of our approach, demonstrating comparable or even stronger results than previous specialist or visual foundation models. We hope our work inspires others to rethink the unification of segmentation and generation.
Problem
Research questions and friction points this paper is trying to address.
Explores in-context segmentation using latent diffusion models.
Develops strategies for instruction extraction and output alignment.
Creates a new benchmark for image and video segmentation.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Utilizes latent diffusion models for segmentation
Implements two-stage masking strategy
Introduces augmented pseudo-masking target
🔎 Similar Papers
No similar papers found.
