Interactive Reward Agent: GUI Task Evaluation via Environment-State Verification

2026-07-28

AI Summary

Automated evaluation of GUI task completion remains challenging due to its reliance on environmental states beyond interface screenshots, such as system configurations and file contents. This work proposes the Interactive Reward Agent (IRA), which introduces environment state verification into GUI evaluation for the first time through a "propose-and-verify" framework. IRA dynamically gathers multi-source evidence by invoking system commands, application APIs, and GUI tools, enabling holistic assessment by fusing visual interface data with underlying environmental information. The authors release GUI-RewardBench, a new benchmark comprising 321 annotated trajectories, on which IRA achieves an accuracy of 86.9%, substantially outperforming existing methods. When employed to train GUI agents via reinforcement learning, IRA attains a 34.0% success rate on OSWorld.

Abstract

Graphical user interface task evaluation aims to determine whether a GUI agent has successfully completed a user instruction. Automated GUI task evaluation has received increasing attention because the evaluation results can serve as reward signals for both test-time scaling and post-training. However, reliable GUI task evaluation remains challenging because the judgments often require access to environment states, such as system configurations, file data, and application settings, beyond the screenshots of execution trajectories. In this paper, we propose an interactive reward agent (IRA) based on a propose-then-verify framework to acquire and verify evidence from the post-execution environment. Given a task instruction and a GUI environment after the GUI agent execution, IRA first proposes the task completion conditions and then verifies them by invoking system tools, application tools, and GUI tools. This design combines evidence from both visible interfaces and the environment state in an interactive process. We further introduce GUI-RewardBench, a benchmark of 321 GUI task trajectories spanning 10 Ubuntu desktop application categories. Experiments show that IRA achieves 86.9% accuracy on GUI-RewardBench, outperforming existing evaluator baselines. We further apply IRA to reinforcement learning of GUI agents, achieving a 34.0% OSWorld success rate, which demonstrates that IRA can provide effective reward signals for training GUI agents.

Problem

• GUI task evaluation
• reward signal
• environment-state verification
• automated evaluation
• interactive agent

Innovation

• Interactive Reward Agent
• propose-then-verify
• environment-state verification
• GUI task evaluation
• reward signal