Digital Twin-Guided Robot Path Planning: A Beta-Bernoulli Fusion with Large Language Model as a Sensor

2025-09-24

AI Summary

This paper addresses the lack of contextual semantic understanding and natural language interaction capability in robotic path planning for construction environments. We propose a Bayesian potential field navigation method that integrates large language models (LLMs) with Building Information Modeling (BIM)-enabled semantic maps. Our key contribution is the first formulation of LLM outputs as pseudo-counts embedded within a Beta-Bernoulli Bayesian fusion framework, enabling principled quantification of uncertainty in natural language instructions. This uncertainty estimate dynamically modulates the obstacle repulsion gain in the potential field method, supporting multi-turn instruction chaining and context-aware path optimization grounded in construction semantics. Experiments demonstrate significant improvements over baselines in path safety, robustness, and task compliance, yielding navigation trajectories that better adhere to real-world construction constraints.

Technology Category

Planning, Routing, and Scheduling: Planning with Language ModelsNatural Language Processing: Safety and RobustnessIntelligent Robots: Motion and Path Planning

Application Category

Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsSearch and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved informationUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Integrating natural language (NL) prompts into robotic mission planning has attracted significant interest in recent years. In the construction domain, Building Information Models (BIM) encapsulate rich NL descriptions of the environment. We present a novel framework that fuses NL directives with BIM-derived semantic maps via a Beta-Bernoulli Bayesian fusion by interpreting the LLM as a sensor: each obstacle's design-time repulsive coefficient is treated as a Beta(alpha, beta) random variable and LLM-returned danger scores are incorporated as pseudo-counts to update alpha and beta. The resulting posterior mean yields a continuous, context-aware repulsive gain that augments a Euclidean-distance-based potential field for cost heuristics. By adjusting gains based on sentiment and context inferred from user prompts, our method guides robots along safer, more context-aware paths. This provides a numerically stable method that can chain multiple natural commands and prompts from construction workers and foreman to enable planning while giving flexibility to be integrated in any learned or classical AI framework. Simulation results demonstrate that this Beta-Bernoulli fusion yields both qualitative and quantitative improvements in path robustness and validity.

Problem

• Integrating natural language prompts with BIM data for robot path planning
• Developing context-aware obstacle avoidance using LLM as semantic sensor
• Creating stable fusion method for construction worker commands in robotics

Innovation

• Beta-Bernoulli Bayesian fusion of LLM and BIM
• LLM interpreted as a sensor for danger scores
• Context-aware repulsive gains for path planning