When Interfaces Speak: Data-Aware Generative UI Harness for Active Interaction

2026-10-07

AI Summary

This study addresses the challenges of cognitive overload and low efficiency in text-based interactions by proposing GenUI-Harness, a multi-agent framework that coordinates tool agents with GUI code generation agents to enable data-aware dynamic interface generation for task execution. Furthermore, it introduces Dynamic UX, a lightweight sandbox environment, alongside a Reward Auditor meta-reward mechanism to effectively mitigate reward hacking in reinforcement learning. Experimental results demonstrate that the proposed approach improves Pass@3 by 4.48%, enabling smaller models to outperform frontier large language models while reducing the average number of dialogue turns from 3.4 to 1.2, thereby substantially enhancing interaction efficiency.

Abstract

Most human-agent interaction today remains text-based. Natural language can impose cognitive overload, ambiguity, information chaos, and slow input for complex tasks; ephemeral generative UIs can present structured information and guide users toward task completion. We propose GenUI-Harness, a multi-agent harness pairing a Tool Agent for information retrieval and task execution with a GUI Coder Agent that identifies ambiguities and generates front-end code for structured interfaces. Training the coder with reinforcement learning is challenging: verifiable rewards for interactive UI generation require costly execution, while LLM-as-a-Judge rewards are prone to reward hacking. We address the first challenge with Dynamic UX, a lightweight package for dynamic interaction and reward collection in a single sandbox, and the second with Reward Auditor, a meta-reward mechanism that monitors reward distributions and distills diagnostic patterns into a shared rubric and scoring specification. We introduce UI-TAU Bench, a benchmark for active human-agent interaction through generated UI code, built on 10 real-world domain databases constructed from public data sources and based on Tau-Bench tool-use settings, with Lite (300 tasks) and Full (1,000 tasks) splits. GenUI-Harness achieves an average Pass@3 gain of 4.48 percentage points over smolagents on Lite. Training with GenUI-Harness improves a 4B backbone from 9.33% to 58.00% Pass@3, outperforming larger frontier models such as Claude Opus 5 (46.67%). GenUI-Harness also remains robust on ambiguous and non-ambiguous queries. In a reviewer survey comparing communication channels, generated UIs reduce average dialogue rounds from 3.4 to 1.2. These results show that data-aware generative interfaces can support effective task completion and reduce dialogue rounds in evaluated database-backed workflows.

Problem

• Human-Agent Interaction
• Generative UI
• Reinforcement Learning
• Reward Hacking
• Cognitive Overload

Innovation

• Generative UI
• Multi-agent system
• Reinforcement learning
• Meta-reward mechanism
• Human-agent interaction