Prior-SG: Task and Prior Driven Region Segmentation for Scene Graphs in Arbitrarily-Structured Environments

2026-08-06

AI Summary

This work addresses the challenge of accurately segmenting semantic regions in open-ended environments for scene graph construction, a task where existing methods falter due to their reliance on local visual clustering or rigid geometric heuristics. The proposed approach formulates scene graph generation as a probabilistic alignment problem, integrating RGB-D instance maps with dynamic task- and structure-level priors derived from large language models (LLMs). By optimizing a Markov random field that fuses these multimodal sources, the method infers high-level functional semantics. Notably, it introduces task-driven LLM-derived topological priors, enabling zero-shot ontological flexibility and supporting task-adaptive spatial reconceptualization even in the absence of physical partitions. Evaluated in both simulated and real-world open environments, the approach achieves state-of-the-art performance and robustly identifies distant functional boundaries.

Abstract

Hierarchical 3D scene graphs are a promising representation for high-level spatial reasoning in autonomous mobile platforms. However, existing extraction frameworks typically rely on purely local visual clustering or strict geometric heuristics, such as wall-separated rooms, which fail in open-plan or arbitrarily-structured environments. We propose Prior-SG, a task- and prior-driven framework that casts scene graph generation fundamentally as a probabilistic alignment problem. As the robot explores, it continuously aggregates an incoming RGB-D sensor stream into a physically grounded Instance Graph utilizing a multi-scale, open-vocabulary feature fusion strategy. The system then infers the high-level functional semantics of this map through a Maximum A Posteriori (MAP) estimate, guided by a Prior Graph-a logical expectation of the environment's structure and task-relevant vocabulary synthesized dynamically by a Large Language Model. By optimizing a Markov Random Field that fuses heterogeneous experts (visual, geometric, and discrete objects) with these topological priors, the system resolves local perceptual ambiguities. We validate this approach across diverse simulated residential datasets and large, open-plan real-world environments. Prior-SG achieves state-of-the-art semantic region segmentation accuracy compared to recent baselines, robustly delineates distant functional boundaries in the absence of physical walls, and uniquely provides zero-shot ontological flexibility, enabling the robot to entirely restructure its spatial partitioning based on a given high-level task.

Problem

• scene graphs
• arbitrarily-structured environments
• semantic region segmentation
• spatial reasoning
• open-plan environments

Innovation

• scene graph generation
• prior-driven segmentation
• zero-shot ontological flexibility
• Large Language Model (LLM) priors
• Markov Random Field fusion