Extending a Large View Synthesis Model for Multi-view Panoptic Segmentation

2026-07-22

AI Summary

This work proposes a high-quality multi-view panoptic segmentation method that operates without explicit 3D reconstruction or task-specific training. By encoding panoptic labels from input views into binary channels and leveraging a pre-trained large-scale view synthesis model for cross-view label propagation, the approach achieves zero-shot label transfer with a frozen model. It represents the first extension of large view synthesis models from appearance rendering to 3D scene understanding, employing cross-view attention mechanisms to ensure label consistency across perspectives. On ScanNet, the method attains segmentation quality comparable to state-of-the-art Gaussian-based 3D reconstruction approaches, while surpassing them by over 7 dB in novel view synthesis metrics. Furthermore, it demonstrates superior zero-shot transfer performance on Replica, outperforming existing methods without any fine-tuning.

Abstract

Large view synthesis models synthesize novel views through cross-view attention without explicit 3D representations, and recent studies have shown that they learn accurate spatial correspondence from RGB supervision alone. We observe that this correspondence generalizes beyond appearance. When non-photorealistic signals such as binary encoded panoptic labels are passed through the model, they are propagated to novel views with consistent spatial structure. These results indicate that the correspondence learned for RGB view synthesis can also propagate view-independent per-pixel labels. From this observation, we present the first work to extend large view synthesis models beyond appearance rendering to 3D scene understanding. We propose a panoptic segmentation pipeline that reuses a frozen view synthesis model to propagate panoptic labels from input views to novel views, without 3D reconstruction or any segmentation-specific training of the view synthesis model. Given panoptic labels on the input views, we encode them into binary channel representations and pass them through the same model to render target-view segmentation. On ScanNet, our method achieves segmentation quality on par with Gaussian based approaches requiring explicit 3D reconstruction, while outperforming them in novel view synthesis by more than 7 dB. The label propagation also transfers across datasets, surpassing these approaches on Replica without any fine-tuning.

Problem

• multi-view panoptic segmentation
• view synthesis
• label propagation
• 3D scene understanding
• novel view synthesis

Innovation

• view synthesis
• panoptic segmentation
• cross-view attention
• 3D scene understanding
• label propagation