UniMPR: A Unified Framework for Multimodal Place Recognition with Arbitrary Sensor Configurations

2025-12-20

AI Summary

This work addresses three key challenges in multimodal place recognition (MPR) under GPS-denied conditions: difficulty in dynamic modality adaptation, poor robustness to modality dropout, and weak cross-platform generalization. To this end, we propose the first MPR framework based on a unified polar-coordinate bird’s-eye-view (polar BEV) feature space. Methodologically, we design a multi-branch heterogeneous encoder with an adaptive fusion architecture, enabling dynamic addition or removal of arbitrary sensor modalities (camera, LiDAR, radar). We further introduce polar BEV feature alignment, adaptive label assignment for pretraining, and multi-source joint training. Evaluated on seven benchmark datasets, our approach achieves state-of-the-art performance, significantly improving recognition accuracy and generalization capability under complex scenes, partial modality failure, and heterogeneous sensor configurations.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Place recognition is a critical component of autonomous vehicles and robotics, enabling global localization in GPS-denied environments. Recent advances have spurred significant interest in multimodal place recognition (MPR), which leverages complementary strengths of multiple modalities. Despite its potential, most existing MPR methods still face three key challenges: (1) dynamically adapting to arbitrary modality inputs within a unified framework, (2) maintaining robustness with missing or degraded modalities, and (3) generalizing across diverse sensor configurations and setups. In this paper, we propose UniMPR, a unified framework for multimodal place recognition. Using only one trained model, it can seamlessly adapt to any combination of common perceptual modalities (e.g., camera, LiDAR, radar). To tackle the data heterogeneity, we unify all inputs within a polar BEV feature space. Subsequently, the polar BEVs are fed into a multi-branch network to exploit discriminative intra-model and inter-modal features from any modality combinations. To fully exploit the network's generalization capability and robustness, we construct a large-scale training set from multiple datasets and introduce an adaptive label assignment strategy for extensive pre-training. Experiments on seven datasets demonstrate that UniMPR achieves state-of-the-art performance under varying sensor configurations, modality combinations, and environmental conditions. Our code will be released at https://github.com/QiZS-BIT/UniMPR.

Problem

• Adapting to arbitrary modality inputs within a unified framework
• Maintaining robustness with missing or degraded sensor modalities
• Generalizing across diverse sensor configurations and setups

Innovation

• Unified framework adapts to arbitrary sensor modality combinations
• Polar BEV feature space unifies heterogeneous multimodal inputs
• Multi-branch network extracts intra-modal and inter-modal features