MissClick: Exploiting Digit-Serialized Coordinates to Attack GUI Grounding Models

2026-08-04

AI Summary

This work addresses a critical security vulnerability in existing GUI visual grounding models, which serialize screen coordinates into numeric tokens whose positional-value structure can be exploited to maliciously shift click locations. To tackle this issue, the authors propose MissClick, a white-box adversarial attack that, for the first time, incorporates the place-value semantics of coordinate digits into a target-oriented loss function. MissClick enables both untargeted disruption (MissClick-U) and targeted hijacking (MissClick-T). Evaluated on the OS-Atlas and UGround benchmarks, MissClick-U achieves attack success rates of 75.07% and 72.93%, respectively, while MissClick-T attains 44.86% and 62.67%, substantially outperforming existing methods and demonstrating its effectiveness and novelty.

Abstract

Recent GUI visual grounding models generate screen coordinates as sequences of digit tokens that are parsed into numerical values and mapped to executable clicks. The security implications of this coordinate generation process have been largely overlooked. We observe that each coordinate digit is predicted as a categorical token, yet after parsing, changing a hundreds-place digit by one changes the corresponding numerical coordinate component by 100 units, which can induce a large displacement of the executed click. This observation motivates attack objectives that account for the numerical and place-value structure of coordinate outputs rather than treating them as ordinary text. Moreover, untargeted and targeted attacks impose different success conditions--displacing the click outside the correct region versus into an attacker-specified region--and therefore benefit from different objectives. We propose MissClick, a simple and effective white-box adversarial attack with two goal-specific objectives: MissClick-U maximizes soft-coordinate displacement for untargeted disruption, while MissClick-T minimizes a place-weighted target-digit loss for targeted hijacking. Compared with existing attacks against GUI grounding models on OS-Atlas and UGround across desktop, web, and mobile platforms, MissClick-U achieves untargeted success rates of 75.07\% and 72.93\% (+16.62 and +30.72 pp), and MissClick-T achieves targeted success rates of 44.86\% and 62.67\% (+31.73 and +47.06 pp). Attack objective comparison further shows that soft-coordinate displacement yields the highest untargeted attack success rate, whereas place-weighted target-digit optimization yields the highest targeted attack success rate, revealing distinct objective preferences for the two attack goals.

Problem

• GUI grounding
• adversarial attack
• coordinate serialization
• place-value vulnerability
• click displacement

Innovation

• digit-serialized coordinates
• GUI grounding attacks
• place-value aware optimization
• soft-coordinate displacement
• targeted adversarial attack