browser automation

Web applications exhibit high interface dynamism and complex navigation and form interactions, posing significant challenges for end-to-end test case generation—particularly low coverage and poor robustness. To address this, we propose a navigation modeling approach that integrates screen transition graphs with large language models (LLMs) to enable high-precision path exploration. We further design a state-graph-based automated testing framework for conditional forms, supporting dynamic DOM parsing and interaction modeling. Additionally, we construct the first dedicated benchmark dataset for form-interaction testing. Experimental evaluation across diverse, complex web applications demonstrates substantial improvements in test coverage and path discovery accuracy—especially for branching navigation and conditional form-filling tasks. Our approach establishes a scalable, empirically evaluable paradigm for web application reliability testing.

Existing automated tools struggle to detect dynamic accessibility issues that depend on user interaction—such as keyboard traps and focus loss—because these problems manifest only at runtime and typically require manual evaluation. This work proposes a novel approach that executes natural language–described interaction sequences within real browsers, records the resulting runtime execution traces, and constructs criterion-specific evidence bundles aligned with WCAG guidelines. Crucially, it introduces an evidence calibration mechanism, establishing runtime interaction traces as the core basis for dynamic accessibility assessment for the first time. Experimental results on 19 real-world web pages evaluating 45 dynamic WCAG criteria demonstrate that the proposed method achieves over a tenfold improvement in Oracle consistency compared to generic browser agents, increases fault precision from 23.5% to 41.4%, and effectively eliminates spurious evidence references.

This work proposes an end-to-end agent framework that addresses the limitations of traditional web usability evaluation, which relies on time-consuming user studies and expert reviews ill-suited for rapid iterative development. The framework uniquely integrates multimodal GUI perception with simulated user behavior profiling to interact directly with live web pages without requiring DOM parsing. It incorporates structured usability protocols—including the System Usability Scale (SUS), Single Ease Question (SEQ), and think-aloud methods—to automatically generate standardized user experience reports. Built upon the Avenir-Web architecture, the approach leverages joint visual-semantic modeling and multimodal action grounding to significantly enhance the automation and scalability of usability testing, thereby empowering developers to efficiently create highly usable web interfaces.

Existing LLM-driven automation methods exhibit significant limitations in execution consistency, precise identification of critical DOM elements, and evaluability when applied to poorly designed, structurally irregular web interfaces within enterprise intranets. To address these challenges, this paper proposes: (1) a standardized operational workflow generation mechanism that reliably transforms demonstrations into robust, executable instructions; (2) a high-precision HTML element localization model integrating both semantic and structural features; and (3) a behavior-trajectory-based quantitative evaluation framework for measuring execution consistency. Evaluated on an internal benchmark, our approach improves task success rate from 72.0% to 88.68% and achieves 84.7% accuracy in operation pattern recognition. These advances substantially enhance the stability, interpretability, and assessability of AI agents in real-world industrial environments.

Existing large language models generate HTML pages that render well statically but often fail under user interactions such as scrolling, clicking, or window resizing—issues largely undetectable by conventional screenshot-based evaluation. This work proposes HTMLCure, a novel framework that leverages browser interaction trajectories as the basis for both assessment and repair. By executing pages across multiple viewports and interactive states, recording deterministic behavioral evidence, and employing a vision-language model to analyze key-frame trajectories, HTMLCure drives a state-aware, closed-loop repair engine. The approach substantially enhances the interactive robustness and usability of generated web pages. Built upon 97K raw prompts, the authors curate a high-quality supervised fine-tuning dataset of 40K samples. The refined HTMLCure-27B-Refined model achieves a score of 50.6 on HTMLBench-400 (with a 45.2% deterministic test pass rate) and 81.2 on MiniAppBench, outperforming baselines by 15.3 points on average.

Existing approaches struggle to evaluate large language models’ generated interactive webpages efficiently and holistically, as they rely either on manual assessment or rigid metrics that fail to capture human-like holistic judgments of dynamic user experiences. This work proposes a reference-free, autonomous holistic evaluation paradigm: we introduce Cookie-Bench, a WebDev benchmark comprising 1,000 prompt-robust queries spanning 11 domains, and design a three-stage framework grounded in Flavell’s metacognitive theory. The framework integrates static perception, agent-driven interaction, and multimodal evidence—including videos, audio, and screenshots—to deliver dual-dimensional scores for both aesthetics and functionality, along with structured attribution. Experiments demonstrate strong alignment with expert ratings and reveal substantial room for improvement among 13 state-of-the-art models in generating interactive web content.

This work proposes an AI-driven autonomous testing framework that addresses the fragility of traditional web test scripts—often susceptible to UI changes and timing issues—and their inability to support natural language–guided security testing. By leveraging a containerized, decoupled architecture, the framework automatically translates natural language instructions into robust web interaction scripts while integrating OWASP Top 10 security validations. It introduces a novel semantic mapping from natural language to security probes and enhances reliability through five key strategies: context-aware selector generation, intelligent wait injection, failure-based learning, and others. Experimental results demonstrate substantial improvements: script generation success rates increase from 55% to 93%, navigation failures decrease eightfold, timing-related race conditions are reduced by 80%, and test authoring time is cut by 75%. Furthermore, security testing achieves detection rates of 85% for authentication bypasses and 95% for input validation flaws, with a false positive rate below 12%.

This study addresses the growing prevalence of bot detection in browser automation, which leads to systematic sample loss in web security and privacy research. Through large-scale measurements across 10,000 websites (40,000 visits) under four browser configurations, this work constructs a taxonomy of bot detection mechanisms and reveals that 82% of access denials are attributable to such detection, with Cloudflare and Akamai accounting for 37% and 26%, respectively. The authors propose a conditional-dependence-based inference method to uncover undeclared detection behaviors, combining custom instrumentation, header spoofing, and cross-configuration comparison. Their analysis shows that HTTP header signals alone explain 75% of the blocks unique to Chromium’s headless mode. Notably, 83% of recent top-tier publications fail to report this bias, underscoring its substantial threat to research validity.

Traditional A/B testing in e-commerce relies on real user traffic, resulting in prolonged experiment cycles and potential degradation of user experience. This work proposes SimGym, a large language model (LLM)-driven browser agent framework that constructs high-fidelity synthetic buyers by extracting user personas and intents from production data, enabling offline simulation of their interactions under both control and treatment conditions. SimGym represents the first approach to integrate LLM-powered agents with real-world user behavioral patterns, effectively replicating the impact of UI changes without involving actual users. Validated on a major e-commerce platform, SimGym reduces experiment duration from weeks to under an hour—even without alignment fine-tuning—demonstrating substantial gains in testing efficiency and scalability.

This work addresses the high learning cost of complex graphical user interfaces and the limitations of existing assistance methods, which often rely on separate chat windows or require extensive custom development. The authors propose an in-situ assistance paradigm that leverages a browser extension to perform lightweight, reversible interventions on the DOM, enabling dynamic interface restructuring without altering the underlying application logic. They introduce the first DOM-based design space and computational pipeline for in-situ assistance, integrating natural language understanding, UI element localization, and reversible operations to support real-time injection of hints, highlighting of controls, or layout rearrangements on arbitrary web pages. Evaluations on two complex interfaces demonstrate the approach’s efficacy and reliability, with user studies showing significant improvements over the ChatGPTAtlas baseline in both usability and task completion efficiency.