Bootstrapped LLM Semantics for Context-Aware Path Planning

2025-11-14

AI Summary

This paper addresses the challenge of safe and efficient path planning for robots operating in semantically rich, human-centric environments under natural language instructions. We propose a risk-aware semantic navigation framework that shifts focus from *what* task to perform—typical of existing methods—to *how* to execute it safely. Specifically, we model large language models (LLMs) as stochastic semantic sensors, leveraging multi-turn prompting and Bayesian bootstrapping to infer category-level risk posterior distributions. These distributions are integrated into a potential field–based cost function. Coupled with a semantic map and a classical planner, the framework enables context-aware, dynamic path reconfiguration. Evaluated in both simulation and BIM-based digital twin scenarios, our approach simultaneously satisfies explicit linguistic directives and implicit environmental constraints, significantly improving path safety and adaptability. Quantitative evaluation demonstrates measurable performance gains over baseline methods.

Technology Category

Natural Language Processing: Safety and RobustnessPlanning, Routing, and Scheduling: Planning with Language ModelsIntelligent Robots: Motion and Path Planning

Application Category

Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactionsSearch and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved informationUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Prompting robots with natural language (NL) has largely been studied as what task to execute (goal selection, skill sequencing) rather than how to execute that task safely and efficiently in semantically rich, human-centric spaces. We address this gap with a framework that turns a large language model (LLM) into a stochastic semantic sensor whose outputs modulate a classical planner. Given a prompt and a semantic map, we draw multiple LLM "danger" judgments and apply a Bayesian bootstrap to approximate a posterior over per-class risk. Using statistics from the posterior, we create a potential cost to formulate a path planning problem. Across simulated environments and a BIM-backed digital twin, our method adapts how the robot moves in response to explicit prompts and implicit contextual information. We present qualitative and quantitative results.

Problem

• Addresses safe robot task execution in human-centric environments using natural language
• Converts LLM judgments into risk-aware path planning through Bayesian bootstrap
• Enables robots to adapt movement based on explicit prompts and contextual semantics

Innovation

• LLM generates stochastic semantic sensor outputs
• Bayesian bootstrap approximates posterior risk classes
• Potential cost formulation enables adaptive path planning