InterOCF: Spatio-Temporal 2D-3D Interaction for Camera-Only 4D Occupancy Forecasting

2026-07-27

AI Summary

Existing methods for 4D occupancy forecasting relying solely on multi-view images struggle to adequately model inter-frame spatiotemporal dynamics, limiting their performance. This work proposes InterOCF, a novel framework that introduces, for the first time, an explicit spatiotemporal interaction mechanism between 2D and 3D branches to jointly model the temporal evolution of 3D voxel sequences and multi-view 2D segmentation sequences. The framework incorporates a 3D spatiotemporal module (3DST), a 2D spatiotemporal module (2DST), and a spatiotemporal interaction modeling module (STIM), effectively fusing voxel-level dynamic predictions with multi-view semantic temporal cues. Additionally, multi-view semantic segmentation is leveraged as an auxiliary task to enhance representational consistency. Experiments demonstrate that InterOCF significantly outperforms existing approaches on the nuScenes, Lyft-Level5, and nuScenes-Occupancy benchmarks.

Abstract

Camera-only 4D occupancy forecasting enables autonomous vehicles to predict future 3D semantic scenes solely from historical multi-view images, which is critical for driving safety. Even though current methods have achieved good performance, the strong spatial-temporal modeling between the input multi-view frames is still underexplored, which limits the performance of those methods in future 4D forecasting. To address this gap, we introduce a novel framework, InterOCF, for 4D occupancy forecasting that jointly models temporal dynamics in both 3D voxel-based representations and multi-view segmentation sequences, while explicitly incorporating feature interaction between the 2D and 3D branches. Our framework incorporates three core components: 1) A 3D Spatio-Temporal (3DST) module that learns volumetric dynamics from historical voxel states to predict future voxel states; 2) A 2D Spatio-Temporal (2DST) module employing an auxiliary multi-view temporal segmentation forecasting task to enhance temporal semantic dynamics; 3) A Spatio-Temporal Interaction Modeling (STIM) module that enables feature interaction between 2D and 3D representations. Experiments on the nuScenes, Lyft-Level5, and nuScenes-Occupancy datasets show that InterOCF consistently outperforms existing baseline approaches.

Problem

• 4D occupancy forecasting
• camera-only perception
• spatio-temporal modeling
• 2D-3D interaction
• autonomous driving

Innovation

• 4D occupancy forecasting
• spatio-temporal interaction
• 2D-3D fusion
• camera-only perception
• temporal dynamics modeling