PromptDet: A Lightweight 3D Object Detection Framework with LiDAR Prompts

2024-12-17

AI Summary

To address inaccurate depth estimation, weak ranging capability, and high training overhead in vision-only multi-camera 3D detection, this paper proposes a lightweight multi-modal bird’s-eye-view (BEV) detection framework. The core innovation is a LiDAR prompt module containing only a minimal number of trainable parameters: during training, it leverages point cloud priors to guide BEV feature learning via cross-modal feature modulation and unidirectional knowledge distillation for enhanced depth perception; at inference, it operates without LiDAR input and automatically reverts to a vision-only mode. The module is plug-and-play with <2% parameter overhead. On nuScenes, the multi-modal variant achieves +22.8% mAP and +21.1% NDS gains over the baseline, while the vision-only inference mode still yields +2.4% mAP and +4.0% NDS improvements, with negligible inference latency increase.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Multi-camera 3D object detection aims to detect and localize objects in 3D space using multiple cameras, which has attracted more attention due to its cost-effectiveness trade-off. However, these methods often struggle with the lack of accurate depth estimation caused by the natural weakness of the camera in ranging. Recently, multi-modal fusion and knowledge distillation methods for 3D object detection have been proposed to solve this problem, which are time-consuming during the training phase and not friendly to memory cost. In light of this, we propose PromptDet, a lightweight yet effective 3D object detection framework motivated by the success of prompt learning in 2D foundation model. Our proposed framework, PromptDet, comprises two integral components: a general camera-based detection module, exemplified by models like BEVDet and BEVDepth, and a LiDAR-assisted prompter. The LiDAR-assisted prompter leverages the LiDAR points as a complementary signal, enriched with a minimal set of additional trainable parameters. Notably, our framework is flexible due to our prompt-like design, which can not only be used as a lightweight multi-modal fusion method but also as a camera-only method for 3D object detection during the inference phase. Extensive experiments on nuScenes validate the effectiveness of the proposed PromptDet. As a multi-modal detector, PromptDet improves the mAP and NDS by at most 22.8% and 21.1% with fewer than 2% extra parameters compared with the camera-only baseline. Without LiDAR points, PromptDet still achieves an improvement of at most 2.4% mAP and 4.0% NDS with almost no impact on camera detection inference time.

Problem

• Multi-camera 3D Object Detection
• Efficiency and Accuracy Improvement
• Resource Optimization

Innovation

• PromptDet
• Fusion Perception
• Efficient 3D Object Detection