Just A Rather Very Intelligent Spoken Agent

2026-07-17

AI Summary

Existing long-horizon AI agents lack continuous, real-time voice interaction mechanisms, hindering users’ ability to understand agent states or intervene promptly. This work proposes an always-on, Jarvis-style voice-mediated system that enables real-time question-answering and proactive status reporting without interrupting the primary task, while seamlessly integrating user guidance into the execution pipeline. We introduce JarvisBench, the first benchmark for evaluating mediation capabilities in long-horizon agent workflows, assessing both collaborative efficacy and user interaction along with a modular reference implementation. Experiments demonstrate that the mediator delivers context-aware responses grounded in execution trajectories, and that even sparse user instructions significantly improve task completion rates. Performance is highly dependent on the underlying large language model, with notable variations observed across models such as GPT-5.5, Claude Opus 4.7, and Gemini.

Abstract

Long-horizon AI agents are becoming increasingly capable, yet their interaction with users remains surprisingly thin. In most workflows, users give an initial instruction, receive only selective textual updates, and lose a clear sense of what the agent is doing or when to step in. This leaves a missing part in the current agent ecosystem: an always-on Jarvis-style mediator that keeps the agent continuously reachable to the user. Such a mediator should support real-time spoken interaction with the user, answer questions without interrupting the worker, proactively report progress or confusion, and inject user guidance back into the agent's execution when useful. In this work, we introduce JarvisBench, a benchmark for measuring the dual value of mediation in long-horizon agent workflows. JarvisBench contains two complementary tracks: an agent-collaboration track that measures whether mediation improves downstream task completion, and a user-interaction track that measures whether mediation makes ongoing execution more understandable, responsive, and accessible to users. We instantiate the benchmark with a modular reference Jarvis prototype and evaluate it on 34 text-only WildClaw tasks executed in OpenClaw. Preliminary results with GPT-5.5, Claude Opus 4.7, Gemini-based, and GPT-based worker agents suggest that Jarvis-style mediation can provide trace-grounded responses to user questions and improve task performance when sparse user guidance is injected at appropriate moments. The results also show that effectiveness depends strongly on the mediator's LLM brain, highlighting both the promise of this missing middle layer and the need for broader community effort. Demo page https://cchen1436.github.io/jarvis

Problem

• long-horizon AI agents
• user interaction
• spoken agent
• mediation
• agent accessibility

Innovation

• Jarvis-style mediation
• long-horizon AI agents
• real-time spoken interaction
• agent-user collaboration
• JarvisBench