RC-AutoCalib: An End-to-End Radar-Camera Automatic Calibration Network

2025-05-28

AI Summary

To address the challenge of online geometric calibration in radar-camera systems—complicated by sparse height measurements and substantial measurement noise—this paper proposes the first end-to-end differentiable auto-calibration framework. Methodologically, it introduces dual-view feature representation with selective fusion, a multimodal cross-attention matching module, and a noise-robust supervised matcher to enable robust, fully automatic runtime calibration without human intervention. The core contribution lies in the first incorporation of radar height uncertainty modeling directly into the end-to-end network, enabling fully automated, online, pixel-level geometric alignment. Evaluated on the nuScenes benchmark, our method significantly outperforms existing radar-camera and LiDAR-camera calibration approaches, establishing a new state-of-the-art. The source code is publicly available.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Browser and end-point security

Abstract

This paper presents a groundbreaking approach - the first online automatic geometric calibration method for radar and camera systems. Given the significant data sparsity and measurement uncertainty in radar height data, achieving automatic calibration during system operation has long been a challenge. To address the sparsity issue, we propose a Dual-Perspective representation that gathers features from both frontal and bird's-eye views. The frontal view contains rich but sensitive height information, whereas the bird's-eye view provides robust features against height uncertainty. We thereby propose a novel Selective Fusion Mechanism to identify and fuse reliable features from both perspectives, reducing the effect of height uncertainty. Moreover, for each view, we incorporate a Multi-Modal Cross-Attention Mechanism to explicitly find location correspondences through cross-modal matching. During the training phase, we also design a Noise-Resistant Matcher to provide better supervision and enhance the robustness of the matching mechanism against sparsity and height uncertainty. Our experimental results, tested on the nuScenes dataset, demonstrate that our method significantly outperforms previous radar-camera auto-calibration methods, as well as existing state-of-the-art LiDAR-camera calibration techniques, establishing a new benchmark for future research. The code is available at https://github.com/nycu-acm/RC-AutoCalib.

Problem

• Online automatic geometric calibration for radar-camera systems
• Addressing radar data sparsity and height uncertainty
• Improving feature fusion and cross-modal matching robustness

Innovation

• Dual-Perspective representation for feature gathering
• Selective Fusion Mechanism for reliable feature fusion
• Multi-Modal Cross-Attention Mechanism for location correspondences