Multi-Scale Semantic Mapping in Urban Environments via Observation Calibration and Policy Dependence Regularization

2026-09-28

AI Summary

This study addresses the unreliability of semantic mapping in urban environments caused by the mismatch between object scales and observation distances. To this end, it proposes a reinforcement learning-based robust semantic mapping framework. Methodologically, we construct the first instance-level multi-scale mapping dataset tailored for urban scenarios and introduce a category-aware likelihood calibration mechanism to filter unreliable observations. Furthermore, a mutual information regularization term is designed to decouple the redundant coupling between calibration and motion policies, while Pareto optimization is employed to balance conflicts in cross-scale value estimation. This work effectively resolves mapping failures induced by large-scale variations, significantly enhancing the accuracy and robustness of semantic maps in complex urban scenes.

Abstract

Semantic mapping is fundamental to embodied navigation, yet existing methods are developed for indoor environments, where objects exhibit relatively limited scale variation and are observed from a restricted range of viewpoints. Urban environments pose substantially greater challenges: agents must map objects ranging from pedestrians to buildings while navigating large spaces with highly diverse viewing distances. These conditions introduce two key difficulties that existing datasets and methods fail to cover. First, object scale and observation distance can be severely mismatched. For example, small objects may be viewed from far away, whereas large objects may be observed at extremely close range, resulting in unreliable observation likelihoods. Second, objects with substantially different sizes and geometries require distinct mapping behaviors, which are difficult to capture with a single shared value estimator. To investigate these challenges, we introduce a large-scale urban semantic mapping dataset featuring realistic city layouts, high-fidelity rendering, and instance-level annotations spanning multiple object scales. We then propose a category-aware likelihood calibration policy that identifies and alleviates unreliable observations according to object category and viewing distance. Because the calibration and motion policies are optimized toward the same mapping objective, they may learn redundant shortcuts and become excessively coupled. We therefore introduce a mutual-information (MI) regularizer that penalizes their estimated representation dependence and encourages complementary behaviors. To better model heterogeneous mapping strategies across object scales, we further employ category-wise value estimators. We formulate their joint optimization as a Pareto optimization problem to mitigate conflicting gradients across categories.

Problem

• semantic mapping
• urban environments
• multi-scale objects
• observation mismatch
• heterogeneous mapping strategies

Innovation

• Semantic Mapping
• Observation Calibration
• Mutual Information Regularization
• Pareto Optimization
• Multi-Scale