OSMa-Bench: Evaluating Open Semantic Mapping Under Varying Lighting Conditions

2025-03-13

AI Summary

This paper addresses the critical problem of significant performance degradation in Open Semantic Mapping (OSM) under indoor illumination variations. To this end, we introduce the first illumination-robustness benchmark for OSM evaluation. Methodologically, we propose a physics-engine-driven, controllable illumination RGB-D synthetic dataset, coupled with 3D ground-truth reconstruction alignment and a scene-graph-based semantic structural consistency evaluation paradigm; we further integrate an LLM/LVLM-powered automated evaluation pipeline. Our core contributions are: (i) the first systematic quantification of semantic fidelity and structural understanding capabilities of leading OSM models—including ConceptGraphs, BBQ, and OpenScene—across multiple illumination levels; and (ii) empirical validation that semantic accuracy (mAP) degrades by an average of 37.2% under reduced illumination, demonstrating the benchmark’s diagnostic effectiveness and generalizability.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Localization, Mapping, and Navigation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Open Semantic Mapping (OSM) is a key technology in robotic perception, combining semantic segmentation and SLAM techniques. This paper introduces a dynamically configurable and highly automated LLM/LVLM-powered pipeline for evaluating OSM solutions called OSMa-Bench (Open Semantic Mapping Benchmark). The study focuses on evaluating state-of-the-art semantic mapping algorithms under varying indoor lighting conditions, a critical challenge in indoor environments. We introduce a novel dataset with simulated RGB-D sequences and ground truth 3D reconstructions, facilitating the rigorous analysis of mapping performance across different lighting conditions. Through experiments on leading models such as ConceptGraphs, BBQ and OpenScene, we evaluate the semantic fidelity of object recognition and segmentation. Additionally, we introduce a Scene Graph evaluation method to analyze the ability of models to interpret semantic structure. The results provide insights into the robustness of these models, forming future research directions for developing resilient and adaptable robotic systems. Our code is available at https://be2rlab.github.io/OSMa-Bench/.

Problem

• Evaluates semantic mapping algorithms under varying lighting conditions.
• Introduces a benchmark for Open Semantic Mapping (OSM) solutions.
• Analyzes semantic fidelity and structure interpretation in robotic perception.

Innovation

• LLM/LVLM-powered automated evaluation pipeline
• Novel dataset with simulated RGB-D sequences
• Scene Graph method for semantic structure analysis