AgentGUI: An Interface for Observing and Steering Long-Running AI Agents

2026-07-28

AI Summary

This work addresses the challenge of effectively supervising complex, long-running AI agents—a task often beyond human capability—by introducing AgentGUI, a locally deployed graphical user interface that enables real-time observation and intervention in multi-agent systems. AgentGUI is the first system to integrate trajectory visualization with both manual and automated control mechanisms, while maintaining compatibility with mainstream open-source and state-of-the-art agent frameworks to support unified human-AI collaborative oversight. Experimental results demonstrate that users leveraging AgentGUI identify critical trajectory elements 38% faster (p = 0.023), and task completion rates for smaller models improve by up to 34 percentage points.

Abstract

AI agents are increasingly adept at tackling complex, long-running tasks. With the rapid surge of autonomous capabilities, human oversight is systematically lagging behind due to limited human-centered interfacing. Aiming to address this, we introduce AgentGUI, a user-friendly, locally hosted GUI for seamlessly observing and steering AI agents amid multiple concurrent, long-running sessions. AgentGUI features 1) rich agent trajectory visualizations, 2) effective manual and automated steering, and 3) integration with and coordination between open-source and frontier agent frameworks. A controlled user study demonstrates statistically significant reduction in the time it takes to identify key elements from agent traces (38% faster, p = 0.023). In a preliminary experiment, AgentGUI's automated drift prevention feature raises the task completion rate of small local agents by as high as 34pp across a 0.8B--9B model ladder (N=50 runs per model). AgentGUI is publicly available through its project website (https://agent-gui-project.github.io) and open-source repository (https://github.com/eth-medical-ai-lab/agent-gui), along with a demo video (https://youtube.com/watch?v=GSDyxN1gTF0).

Problem

• AI agents
• human oversight
• user interface
• long-running tasks
• autonomous systems

Innovation

• AgentGUI
• agent steering
• trajectory visualization
• drift prevention
• human-AI collaboration