Embodied Agents Take Control: Minimal-Interface Zero-Shot Agents Rival Industrial-Scale Policies in Vision-and-Language Navigation

2026-07-28

AI Summary

This work investigates efficient, zero-shot vision-and-language navigation under a minimal setting using only monocular RGB images and discrete actions. It proposes a general-purpose agent architecture featuring a closed-loop decision-making mechanism—perceive, act, verify, and self-correct—that accomplishes navigation tasks without task-specific fine-tuning or predefined pipelines. Experiments demonstrate that the method achieves up to 78% success rate on Fable-5 in its default configuration; when augmented with an optional waypoint tool, it attains a 76.7% success rate using less than half the environment steps and one-quarter of the wall-clock time required by conventional approaches. This study provides the first evidence that a generalist agent, operating under minimal interfaces and zero-shot conditions, can rival industrial-grade policies, while also revealing the complementary yet distinct roles of model capabilities, agent frameworks, and tool interfaces.

Abstract

Autonomous embodied agents must sustain a long decision-making loop that involves perceiving, acting, verifying, and self-correcting over many steps. Current systems sustain this loop through task-specific workflows or embodied policies. We study a third form, agentic embodied control, in which a general-purpose agent holds the loop itself. Using zero-shot navigation as a controlled testbed, we evaluate three software-engineering agent harnesses given only a monocular RGB camera and discrete actions. Under this strictly minimal condition, replicated default-effort configurations reach 70.7$\pm$3.5% success (opus-5, mean over three runs), and fable-5 reaches 78% at maximum effort. When a trained waypoint tool is exposed alongside primitives as an optional capability, the hybrid fable-5 agent reaches 76.7$\pm$0.6% at default effort, using half the environment steps and less than one quarter of the wall time of the maximum-effort primitive run. Controlled interventions show that capability is primarily model-centered: model choice strongly changes success, harness effects are descriptive, and a forced waypoint interface helps weaker models but can hinder stronger ones. Performance nevertheless falls sharply on longer-horizon tasks, while latency and context growth limit sustained operation. These results show that agentic control is already competitive in zero-shot navigation and that models, harnesses, and interfaces offer complementary paths toward autonomous embodied agents.

Problem

• embodied agents
• vision-and-language navigation
• zero-shot control
• autonomous decision-making
• minimal-interface

Innovation

• agentic embodied control
• zero-shot navigation
• minimal-interface agents
• vision-and-language navigation
• general-purpose agent