Tuning-Free Amodal Segmentation via the Occlusion-Free Bias of Inpainting Models

2025-03-24

AI Summary

Current amodal segmentation methods heavily rely on supervised annotations or synthetic data, limiting their generalization capability. To address this, we propose the first zero-shot, fine-tuning-free amodal segmentation framework that simultaneously predicts precise masks for both visible and occluded object regions without any additional training. Our core insight is the novel discovery and exploitation of an inherent “occlusion-free preference” prior embedded in pre-trained diffusion inpainting models. By coupling visibility-guided occlusion reconstruction with segmentation, our method enables end-to-end zero-shot inference. It requires only a standard diffusion model (e.g., Stable Diffusion) and an input image’s visible mask. Evaluated on five benchmark datasets, our approach achieves an average 5.3% improvement in mask accuracy over prior state-of-the-art methods, demonstrating strong cross-domain generalization and robustness.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Amodal segmentation aims to predict segmentation masks for both the visible and occluded regions of an object. Most existing works formulate this as a supervised learning problem, requiring manually annotated amodal masks or synthetic training data. Consequently, their performance depends on the quality of the datasets, which often lack diversity and scale. This work introduces a tuning-free approach that repurposes pretrained diffusion-based inpainting models for amodal segmentation. Our approach is motivated by the "occlusion-free bias" of inpainting models, i.e., the inpainted objects tend to be complete objects without occlusions. Specifically, we reconstruct the occluded regions of an object via inpainting and then apply segmentation, all without additional training or fine-tuning. Experiments on five datasets demonstrate the generalizability and robustness of our approach. On average, our approach achieves 5.3% more accurate masks over the state-of-the-art.

Problem

• Predicting segmentation masks for visible and occluded object regions
• Eliminating reliance on manually annotated or synthetic training data
• Leveraging pretrained inpainting models for tuning-free amodal segmentation

Innovation

• Repurposes pretrained diffusion-based inpainting models
• Utilizes occlusion-free bias for amodal segmentation
• No additional training or fine-tuning required