Jarvis: A Proactive Speech Agent for Multi-Party Conversations

2026-10-05

AI Summary

This study addresses the lack of proactive intervention capabilities in voice agents during multi-party conversations by proposing Jarvis, a real-time proactive voice agent. By monitoring group discussions via a shared document, Jarvis intervenes exclusively when uncorrected factual errors arise. This work formalizes a proactive intervention task grounded in cognitive biases and constructs the CHI-180-proactive dataset. The proposed architecture leverages a compact open-source model integrating deterministic verification with source-sentence attribution, combined with evidence presentation and real-time speech interaction. Experimental results demonstrate that the system accurately corrects errors on synthetic data while remaining silent in 97% of scenarios. Furthermore, a 23-participant real-time user study validates its effectiveness and minimal intrusiveness in authentic multi-party dialogues.

Abstract

Speech agents are reactive and dyadic: they speak when spoken to, and to one person at a time. We ask what it takes for a speech agent to instead take part in a conversation among several people and speak up only when it can help. We introduce Jarvis, a real-time proactive speech agent that audibly participates in multi-party human conversations. Grounded in a document shared beforehand, Jarvis follows the discussion and intervenes when the group misses or misstates a fact and does not correct itself within a few turns. We make three contributions: a problem setting based on epistemic breakdowns that makes proactive intervention measurable, realized as CHI-180-proactive, a synthetic multi-party dataset seeded with known gaps, errors, and self-corrections; a proactive backbone that harnesses a small, open-weight model with deterministic checks and grounds every claim in a source sentence; and interaction techniques for taking the floor in live speech and showing the cited evidence on screen. On CHI-180-proactive, Jarvis is correct on most events it addresses and stays silent 97% of the time when the group resolves an issue itself. A live study with 23 participants confirms these trends with real-time interventions.

Problem

• speech agent
• multi-party conversations
• proactive intervention
• epistemic breakdowns

Innovation

• Proactive Speech Agent
• Multi-Party Conversations
• Epistemic Breakdowns
• Grounded Generation
• Real-time Intervention