How Many Pixels Is a Digit Worth? Place-Aware Coordinate Entropy for GUI Agent Confidence Estimation

2026-09-21

AI Summary

研究解决了GUI代理点击坐标预测的置信度估计问题，通过提出基于位置感知的坐标熵（PACE）方法，提高了正确点击与错误点击的区分能力。

Abstract

GUI agents predict click coordinates as digit-token sequences, but standard text-LLM confidence estimation methods rank correct clicks from wrong ones only weakly. GUI-specific alternatives use K samples or new supervision, but still leave room for improvement. We trace part of this to place-value asymmetry: bounding-box correctness often makes higher-place digits more important than lower-place digits, so uniform aggregation weakens the signal that determines correctness. The fix is to weight each digit's Shannon entropy by its place value. We call this Place-Aware Coordinate Entropy (PACE). Across fixed-scale agents on ScreenSpot-Pro and ScreenSpot-v2, PACE wins both AUROC and selective accuracy on all primary comparisons in a single forward pass, matching or outperforming K-sample baselines at a fraction of the cost. PACE provides a per-click confidence estimate that turns coordinate-token internals into a practical confidence signal for GUI agent deployment.

Problem

• GUI agents
• confidence estimation
• place-value asymmetry
• bounding-box correctness

Innovation

• Place-Aware Coordinate Entropy
• PACE
• GUI Agent
• Confidence Estimation
• Shannon Entropy