Investigating the Effectiveness of a Socratic Chain-of-Thoughts Reasoning Method for Task Planning in Robotics, A Case Study

2025-03-11

AI Summary

This work investigates the feasibility of directly deploying large language models (LLMs) for spatial task planning—specifically, complex object search requiring physical interaction—under zero-shot conditions. Method: We propose SocraCoT, the first framework integrating Socratic questioning into chain-of-thought (CoT) reasoning and augmenting it with EVINCE-LoC to enable spatially aware robot code generation without fine-tuning. Experiments are conducted using GPT-4 Omni within the Webots simulation environment and on the real-world Tiago robot platform. Contribution/Results: Across 20 dynamic scene trials, SocraCoT achieves significantly higher task completion rates and robustness compared to standard CoT and non-reasoning baselines. The approach establishes a new paradigm for lowering robotic programming barriers, enhancing natural human-robot collaboration, and reducing computational overhead—all while operating in a zero-shot setting.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Intelligent Robots: Motion and Path Planning
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large language models (LLMs) have demonstrated unprecedented capability in reasoning with natural language. Coupled with this development is the emergence of embodied AI in robotics. Despite showing promise for verbal and written reasoning tasks, it remains unknown whether LLMs are capable of navigating complex spatial tasks with physical actions in the real world. To this end, it is of interest to investigate applying LLMs to robotics in zero-shot learning scenarios, and in the absence of fine-tuning - a feat which could significantly improve human-robot interaction, alleviate compute cost, and eliminate low-level programming tasks associated with robot tasks. To explore this question, we apply GPT-4(Omni) with a simulated Tiago robot in Webots engine for an object search task. We evaluate the effectiveness of three reasoning strategies based on Chain-of-Thought (CoT) sub-task list generation with the Socratic method (SocraCoT) (in order of increasing rigor): (1) Non-CoT/Non-SocraCoT, (2) CoT only, and (3) SocraCoT. Performance was measured in terms of the proportion of tasks successfully completed and execution time (N = 20). Our preliminary results show that when combined with chain-of-thought reasoning, the Socratic method can be used for code generation for robotic tasks that require spatial awareness. In extension of this finding, we propose EVINCE-LoC; a modified EVINCE method that could further enhance performance in highly complex and or dynamic testing scenarios.

Problem

• Evaluating LLMs for spatial task planning in robotics.
• Testing Socratic Chain-of-Thought reasoning in zero-shot learning.
• Improving robot task execution with minimal programming effort.

Innovation

• GPT-4 with simulated Tiago robot in Webots
• Socratic Chain-of-Thought reasoning for robotics
• EVINCE-LoC for complex dynamic scenarios