Broken Gates: Re-evaluating Web Bot Defenses in the Age of LLM Agents

2026-07-20

AI Summary

This study systematically evaluates the effectiveness of current web anti-bot mechanisms against large language model (LLM)-based browser agents. It examines mainstream interactive CAPTCHAs (hCaptcha, reCAPTCHA v2/v3) and non-interactive trust scoring systems (e.g., Cloudflare Turnstile) against seven commercial solvers and six categories of LLM agents—spanning cloud-hosted, self-hosted, AI-assisted, and browser extension implementations. Through fine-grained interaction trace analysis, the work reveals for the first time that the real-world security of non-interactive defenses hinges primarily on the authenticity of the execution environment rather than user behavioral characteristics. Notably, LLM agents exhibiting highly similar behaviors achieve markedly different bypass success rates due to environmental differences. Findings indicate that interactive CAPTCHAs are nearly perfectly and inexpensively circumvented by commercial solvers, while LLM agents equipped with dedicated solving modules also bypass them effectively; non-interactive schemes offer stronger resistance but rely fundamentally on environment fingerprinting.

Abstract

LLM-based browser agents are rapidly changing the threat landscape for web security. Unlike traditional automation frameworks that execute predefined scripts, these agents can autonomously navigate websites, reason about page content, and interact with web interfaces using natural-language instructions. This evolution raises fundamental questions about the effectiveness of bot management systems, widely deployed to defend against automated web abuse. In this paper, we present a systematic measurement study evaluating the resilience of both interactive challenge-based defenses and non-interactive trust-based defenses against two attacker classes: commercial Captcha-solving services and LLM-based browser agents. Our evaluation spans seven solver services and six agents, including cloud-hosted, self-hosted, AI-assisted, and browser-extension configurations, tested against hCaptcha, reCaptcha v2, reCaptcha v3, and Cloudflare Turnstile. Our results show that challenge-based defenses are broadly ineffective against commercial solvers, which achieve near-perfect bypass at negligible cost. The challenges can similarly be defeated by LLM-based agents when a dedicated solver module is available. Non-interactive defenses such as reCaptcha v3 exhibit stronger resistance, but our analysis reveals that this resilience does not reflect a fundamental security property. Through fine-grained interaction trace analysis, we find that two agents with nearly indistinguishable behavioral footprints yield divergent outcomes, one bypassing the defense and one failing, isolating execution-environment authenticity, rather than agent behavior, as the determining factor. These findings suggest that the security boundary of non-interactive defenses lies at the environment layer, with significant implications for how bot management systems are designed and evaluated.

Problem

• web bot defenses
• LLM agents
• captcha bypass
• bot management
• security evaluation

Innovation

• LLM-based agents
• bot detection
• CAPTCHA bypass
• execution environment authenticity
• web security