Occlusion-Aware Panoptic Segmentation with Joint Position Embedding and Occlusion-Level Attention

2026-07-20

AI Summary

This work addresses the significant performance degradation of existing panoptic segmentation methods in complex occlusion scenarios, primarily due to their inadequate modeling of occlusion relationships. To tackle this limitation, we propose PEMOLA, a novel module that, for the first time, incorporates occlusion-level attention as spatial guidance and encodes occlusion labels into learnable embeddings to generate channel-wise weights. These weights jointly modulate positional embeddings within a Transformer architecture, thereby injecting explicit occlusion priors. We further construct and release two new benchmark datasets—COCO-OLAC and Cityscapes-OLAC—with consistent, cross-dataset occlusion annotations following a unified labeling protocol. Experimental results demonstrate that our approach substantially improves panoptic segmentation quality on both datasets with minimal computational overhead, confirming the critical role of explicit occlusion modeling in enhancing segmentation robustness.

Abstract

Panoptic segmentation in complex scenes remains challenging because of occlusions, yet modern approaches often neglect occlusion modelling. In this paper, we propose \textbf{P}osition \textbf{E}mbedding \textbf{M}odulation with \textbf{O}cclusion-\textbf{L}evel \textbf{A}ttention (PEMOLA), a novel occlusion-aware module that can be seamlessly integrated into transformer-based panoptic segmentation. To obtain occlusion cues, we train an occlusion classifier on the COCO-OLAC dataset. The classifier derives the occlusion-level attention, which serves as spatial guidance, while the occlusion labels are encoded into a learnable embedding to produce channel-wise weights. Through joint modulation, PEMOLA elegantly introduces the occlusion priors into the position embedding, thereby improving the occlusion modelling. We further annotate the Cityscapes dataset with occlusion levels, termed Cityscapes Occlusion Labels for All Computer Vision Tasks (Cityscapes-OLAC), following the same labelling protocol as COCO-OLAC, to evaluate the cross-dataset generalisation ability of PEMOLA. Extensive experiments on COCO-OLAC and Cityscapes-OLAC demonstrate that PEMOLA consistently improves panoptic segmentation quality while introducing minimal computational overhead. These results highlight the importance of occlusion modelling, where incorporating occlusion-level attention helps deliver robust panoptic segmentation under occlusion. Code and dataset are available at https://github.com/wenbo-wei/PEMOLA.

Problem

• occlusion
• panoptic segmentation
• occlusion modelling
• complex scenes
• occlusion-aware

Innovation

• occlusion-aware
• panoptic segmentation
• position embedding modulation
• occlusion-level attention
• transformer-based vision