Learning Adaptive Semantic Gaussian Allocation for 3D Occupancy

2026-07-23

AI Summary

This work addresses the limitation in semantic 3D Gaussian representations where a fixed total number of Gaussians often leads to redundancy in simple regions and insufficient semantic support in complex areas. To overcome this, the paper proposes the Semantic Gaussian Allocation Transformer (SAGFormer), which introduces an explicit capacity allocation mechanism for the first time. SAGFormer leverages a Transformer architecture to fuse Gaussian attributes with local geometric-semantic features, enabling scoring and selection of an optimal subset of candidate Gaussians for semantic occupancy prediction. This approach optimizes the spatial and semantic distribution of Gaussians under a constrained budget, significantly improving occupancy prediction performance on nuScenes-SurroundOcc and SSCBench-KITTI-360. The method yields more consistent semantic representations, stronger class-consistent voxel support, and fewer unused Gaussians.

Abstract

Semantic 3D Gaussians provide a compact representation for 3D semantic occupancy prediction by rendering semantic primitives into a voxel volume under voxel-wise supervision. Recent methods have improved the modeling ability and efficiency of this representation through more flexible primitive shapes, geometry-guided initialization, and progressive densification. However, these advances mainly determine how primitives are represented, initialized, or added, and do not explicitly address how to select the most useful Gaussians when their total number must be limited to control memory and computation. This imbalance creates an allocation bottleneck: redundant Gaussians remain in simple regions, while difficult regions receive insufficient semantic support. We propose the Semantic Gaussian Allocation Transformer (SAGFormer), which uses Gaussian attributes and local geometric-semantic features to score candidates and select a fixed final Gaussian set. Experiments on nuScenes-SurroundOcc and SSCBench-KITTI-360 show that SAGFormer improves occupancy prediction under the evaluated protocols and yields more semantically consistent and better-utilized Gaussian representations. Under similar final counts and raw coverage, it reduces semantic mixing, strengthens class-consistent voxel support, and produces fewer unused Gaussians. The results indicate that explicit capacity allocation is a useful complement to Gaussian refinement for semantic occupancy prediction.

Problem

• Semantic 3D Gaussians
• 3D occupancy prediction
• Gaussian allocation
• capacity allocation
• semantic consistency

Innovation

• Semantic Gaussian Allocation
• 3D Occupancy Prediction
• Adaptive Selection
• SAGFormer
• Capacity-aware Representation