H3O: Hyper-Efficient 3D Occupancy Prediction with Heterogeneous Supervision

2025-03-06

AI Summary

To address the high computational cost and ambiguous ground-truth annotation challenges in 3D occupancy prediction for autonomous driving, this paper proposes a lightweight multi-task 2D-supervised paradigm. Instead of costly 3D attention mechanisms and voxel-based feature processing, our approach jointly optimizes multi-view depth estimation, 2D semantic segmentation, and surface normal prediction to implicitly constrain 3D occupancy reasoning. We introduce differentiable volumetric rendering and a lightweight 2D-to-3D feature mapping module, enabling end-to-end training. Evaluated on Occ3D-nuScenes and SemanticKITTI, our method achieves state-of-the-art accuracy while improving inference speed by 3.2× and reducing FLOPs significantly compared to mainstream approaches. To the best of our knowledge, this is the first work to unify high accuracy and high efficiency in 3D occupancy prediction, offering a practical solution for real-time autonomous driving systems.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: State Estimation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

3D occupancy prediction has recently emerged as a new paradigm for holistic 3D scene understanding and provides valuable information for downstream planning in autonomous driving. Most existing methods, however, are computationally expensive, requiring costly attention-based 2D-3D transformation and 3D feature processing. In this paper, we present a novel 3D occupancy prediction approach, H3O, which features highly efficient architecture designs that incur a significantly lower computational cost as compared to the current state-of-the-art methods. In addition, to compensate for the ambiguity in ground-truth 3D occupancy labels, we advocate leveraging auxiliary tasks to complement the direct 3D supervision. In particular, we integrate multi-camera depth estimation, semantic segmentation, and surface normal estimation via differentiable volume rendering, supervised by corresponding 2D labels that introduces rich and heterogeneous supervision signals. We conduct extensive experiments on the Occ3D-nuScenes and SemanticKITTI benchmarks that demonstrate the superiority of our proposed H3O.

Problem

• Efficient 3D occupancy prediction for autonomous driving
• Reducing computational cost in 3D scene understanding
• Enhancing 3D supervision with auxiliary 2D tasks

Innovation

• Efficient 3D occupancy prediction architecture
• Heterogeneous supervision with auxiliary tasks
• Differentiable volume rendering for 2D-3D integration