MATS: A novel multi-modality multi-task learning framework for 3D perception in autonomous driving

2026-07-27

AI Summary

This work addresses the limitations of existing methods that rely on a single bird’s-eye-view (BEV) feature map, which struggles to meet the diverse requirements of multimodal 3D perception across multiple tasks. To overcome this, we propose the Modality-Adaptive Task-Specific (MATS) framework, which introduces a novel modality-adaptive BEV fusion mechanism to generate tailored feature representations for camera and LiDAR modalities. Furthermore, MATS incorporates task-specific Mixture-of-Experts (MoE) modules to enable decoupled multi-task learning and adaptive feature selection. Evaluated on the nuScenes benchmark, our approach significantly outperforms current state-of-the-art methods, achieving substantial improvements in both single-task and multi-task performance.

Abstract

Multi-modality data from different sensors provides rich complementary information for 3D perception, becoming an essential component in reliable autonomous driving systems. Current research typically designs intricate and complex fusion strategies to integrate information from multimodal data on a unified bird's-eye-view (BEV) feature map for the joint learning of multiple perception tasks. However, such a single feature map hardly carries sufficient information to simultaneously meet the requirements of various perception tasks, leading to a very limited perception performance. To mitigate this limitation, this paper proposes MATS, a novel multi-modality multi-task learning approach with modality-adaptive BEV fusion and task-specific Mixture-of-Experts (MoE) for 3D perception. Specifically, a simple modality-adaptive BEV fusion module is designed to adaptively recalibrate the BEV features by modeling the global cross-modality dependencies, generating diverse BEV feature maps for various perception tasks. For joint multi-task learning, this paper proposes a task-specific MoE module to decouple the tasks and enable the network to automatically choose the appropriate BEV feature candidates for each specific task. To validate the effectiveness of the proposed approach, we conduct extensive experiments on the large-scale benchmark nuScenes. With the camera- and LiDAR-modality input data, the proposed approach outperforms the state-of-the-art (SOTA) by a significant margin. Furthermore, the experimental results on the single tasks show that the proposed approach significantly outperforms the baselines. The code and trained models will be available upon publication.

Problem

• multi-modality
• multi-task learning
• 3D perception
• BEV fusion
• autonomous driving

Innovation

• multi-modality
• multi-task learning
• BEV fusion
• Mixture-of-Experts
• 3D perception