MAESTRO: Task-Relevant Optimization via Adaptive Feature Enhancement and Suppression for Multi-task 3D Perception

2025-09-22

AI Summary

In multi-task 3D perception, inter-task feature interference induces optimization conflicts, hindering joint performance across 3D object detection, BEV map segmentation, and 3D occupancy prediction. To address this, we propose a structured multi-task decoupling framework: (1) a class-level prototype generator explicitly models semantic priors; (2) task-specific feature generators enable adaptive feature enhancement and suppression; and (3) a scene prototype aggregator dynamically fuses cross-task contextual information. Leveraging a shared-separate collaborative representation mechanism, our framework decouples task dependencies while strengthening task-specific feature discrimination at the feature level. Extensive experiments on nuScenes and Occ3D benchmarks demonstrate consistent and significant improvements over state-of-the-art methods across all three tasks, validating the effectiveness of our approach in mitigating task conflicts, enhancing generalization, and improving joint multi-task performance.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

The goal of multi-task learning is to learn to conduct multiple tasks simultaneously based on a shared data representation. While this approach can improve learning efficiency, it may also cause performance degradation due to task conflicts that arise when optimizing the model for different objectives. To address this challenge, we introduce MAESTRO, a structured framework designed to generate task-specific features and mitigate feature interference in multi-task 3D perception, including 3D object detection, bird's-eye view (BEV) map segmentation, and 3D occupancy prediction. MAESTRO comprises three components: the Class-wise Prototype Generator (CPG), the Task-Specific Feature Generator (TSFG), and the Scene Prototype Aggregator (SPA). CPG groups class categories into foreground and background groups and generates group-wise prototypes. The foreground and background prototypes are assigned to the 3D object detection task and the map segmentation task, respectively, while both are assigned to the 3D occupancy prediction task. TSFG leverages these prototype groups to retain task-relevant features while suppressing irrelevant features, thereby enhancing the performance for each task. SPA enhances the prototype groups assigned for 3D occupancy prediction by utilizing the information produced by the 3D object detection head and the map segmentation head. Extensive experiments on the nuScenes and Occ3D benchmarks demonstrate that MAESTRO consistently outperforms existing methods across 3D object detection, BEV map segmentation, and 3D occupancy prediction tasks.

Problem

• Mitigating task conflicts in multi-task 3D perception learning
• Reducing feature interference between different 3D perception tasks
• Enhancing task-specific features while suppressing irrelevant ones

Innovation

• Generates task-specific features via adaptive enhancement
• Uses class-wise prototypes to suppress irrelevant feature interference
• Aggregates scene prototypes to boost 3D occupancy prediction