DiagEval: Trajectory-Conditioned Diagnosis for Reliable Software Evaluation with GUI Agents

Unknown Date

AI Summary

This work addresses a critical limitation in interactive software evaluation: GUI agents often cannot distinguish whether an evaluation failure stems from their own execution errors or genuine software defects, as they rely solely on a single execution trajectory. To resolve this, the authors propose DiagEval, a diagnostic evaluation protocol that leverages failed trajectories to generate targeted probes after an initial failure and aggregates the outcomes of these probes to attribute the cause of failure accurately. Notably, DiagEval is the first method to use failed trajectories to guide probe generation without requiring UI state graph reconstruction or posterior probability estimation. Experimental results demonstrate that DiagEval improves evaluation accuracy to 78.3% on WebDevJudge-Unit and 81.6% on RealDevBench, while successfully recovering 45.6%–62.1% of false-negative cases—substantially outperforming simple retry baselines.

Abstract

Evaluating LLM-generated interactive software requires execution in addition to static analysis. The key difficulty is that correctness is a graph-level reachable property over latent UI state-transition graphs, whereas a GUI evaluator observes only a single execution trajectory. A failed rollout therefore rules out only one realized path, leaving failure attribution ambiguous between evaluator-side execution error and genuine software defect. We present DiagEval, a trajectory-conditioned diagnostic evaluation protocol for post-failure GUI-agent evaluation of interactive software. Rather than blindly retrying from scratch, DiagEval reuses the failed trajectory to choose targeted diagnostic probes and aggregates their outcomes into an internal attribution signal. The latent-graph view motivates the diagnostic problem; DiagEval does not reconstruct the graph or estimate calibrated posterior probabilities. We evaluate DiagEval on WebDevJudge-Unit and RealDevBench across multiple GUI-agent evaluators and LLM backbones. On false-negative cases, DiagEval recovers 45.6-62.1% of failures that were initially misattributed to software defects, outperforming retry-based baselines with 34.4-160.6% relative gains. On the full evaluation sets, this recovery improves accuracy from 69.9% to 78.3% on WebDevJudge-Unit and from 65.0% to 81.6% on RealDevBench. These results suggest that reliable GUI-agent evaluation requires not only stronger execution, but also active failure diagnosis to disambiguate evaluator-side errors from genuine software defects. Our code is available at https://github.com/scutGit/DiagEval.

Problem

• GUI agents
• software evaluation
• failure attribution
• trajectory
• interactive software

Innovation

• trajectory-conditioned diagnosis
• GUI agent evaluation
• failure attribution
• interactive software testing
• diagnostic probing