RoboPrompt: Intuitive Robot Policy Steering with Sparse Human Input

2026-10-07

AI Summary

This study addresses the challenge of zero-shot deployment in robotic policies constrained by limited data, as well as the reliance of conventional shared autonomy on dedicated hardware or architectural modifications. To overcome these limitations, this work proposes a non-intrusive guidance framework that employs a lightweight module to transform sparse human inputs into action drafts, thereby decoupling intent translation from policy execution. Furthermore, it leverages diffusion models to dynamically optimize generation within the noise space, enabling universal guidance across diverse policies without fine-tuning or architectural changes. Experimental results demonstrate that the proposed method improves multi-task success rates by up to 21.3% while reducing human intervention by 81.9%, validating both its effectiveness and scalability.

Abstract

End-to-end robot policies trained through imitation learning remain constrained by limited data diversity, making reliable zero-shot deployment in real-world settings challenging. Shared-autonomy methods enable human correction through teleoperation, but specialized hardware and operator training hinder deployment at scale. Other approaches incorporate human guidance as additional policy inputs, often requiring architectural changes and dedicated training for steerability, which limits their applicability across policies. We present RoboPrompt, a general-purpose, lightweight robot policy steering system that enables users to guide policy behavior through intuitive, sparse inputs, including drawn traces, target points, and coarse directional instructions. RoboPrompt decouples human-intention translation from the underlying policy: a reusable module converts human guidance into action drafts, which are refined through the diffusion or flow-matching dynamics of the base policy. By controlling action generation in noise space, RoboPrompt balances human intent with the policy prior without modifying the base policy architecture or fine-tuning it for steerability. Experiments demonstrate effective steering across Diffusion Policy, $π_{0.5}$, and FastWAM. We further use steered rollouts for online policy improvement through DAgger. After 2-3 rounds of iteration, average success rates increase by 15.5\% for $π_{0.5}$ across three tasks and by 21.3\% across three policies(Diffusion Policy, $π_{0.5}$, FastWAM) on the Insert Bread task, while average human intervention counts decrease by 44.0\% (2.86 to 1.60) and 81.9\% (2.60 to 0.47), respectively.

Problem

• robot policy steering
• imitation learning
• shared autonomy
• human guidance
• zero-shot deployment

Innovation

• Policy Steering
• Sparse Human Input
• Diffusion Policy
• Shared Autonomy
• Online Policy Improvement