Anticipate, Adapt, Act: A Hybrid Framework for Task Planning

2026-02-23

AI Summary

This work addresses the challenge of execution failures in complex human-robot collaborative tasks, which often arise from human capability limitations or missing environmental objects—scenarios that existing approaches struggle to anticipate and mitigate effectively. To overcome this, we propose a novel hybrid framework that integrates large language models (LLMs) with Relational Dynamic Influence Diagram Language (RDDL). This approach uniquely combines the general-purpose semantic reasoning capabilities of LLMs with RDDL’s probabilistic sequential decision-making mechanism, enabling proactive failure prediction and timely intervention in task execution. Evaluated in the VirtualHome 3D simulation environment, our method significantly outperforms current baselines, demonstrating substantial improvements in both task success rate and system robustness.

Technology Category

• Intelligent Robots: Human-Robot Interaction
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Anticipating and adapting to failures is a key capability robots need to collaborate effectively with humans in complex domains. This continues to be a challenge despite the impressive performance of state of the art AI planning systems and Large Language Models (LLMs) because of the uncertainty associated with the tasks and their outcomes. Toward addressing this challenge, we present a hybrid framework that integrates the generic prediction capabilities of an LLM with the probabilistic sequential decision-making capability of Relational Dynamic Influence Diagram Language. For any given task, the robot reasons about the task and the capabilities of the human attempting to complete it; predicts potential failures due to lack of ability (in the human) or lack of relevant domain objects; and executes actions to prevent such failures or recover from them. Experimental evaluation in the VirtualHome 3D simulation environment demonstrates substantial improvement in performance compared with state of the art baselines.

Problem

• failure anticipation
• human-robot collaboration
• task planning
• uncertainty
• adaptive planning

Innovation

• hybrid planning framework
• large language models
• relational dynamic influence diagrams
• failure anticipation and recovery
• human-robot collaboration