Detecting Bot Detection: Prevalence, Techniques, and Implications for Web Measurement Research

2026-06-12

AI Summary

This study addresses the growing prevalence of bot detection in browser automation, which leads to systematic sample loss in web security and privacy research. Through large-scale measurements across 10,000 websites (40,000 visits) under four browser configurations, this work constructs a taxonomy of bot detection mechanisms and reveals that 82% of access denials are attributable to such detection, with Cloudflare and Akamai accounting for 37% and 26%, respectively. The authors propose a conditional-dependence-based inference method to uncover undeclared detection behaviors, combining custom instrumentation, header spoofing, and cross-configuration comparison. Their analysis shows that HTTP header signals alone explain 75% of the blocks unique to Chromium’s headless mode. Notably, 83% of recent top-tier publications fail to report this bias, underscoring its substantial threat to research validity.

Abstract

Browser automation frameworks are essential tools for security and privacy research on the web, yet bot detection scripts increasingly probe their artifacts, threatening measurement validity as automated browsers may be blocked or served different content. Prior work measures detection deployment, while we measure blocking-induced sample loss. Through a literature survey of top-tier security, privacy, and web measurement venues, we find that 83% of papers omit any discussion of bot detection blocking. To address this gap, we conduct a measurement study of 10,000 websites across four browser configurations (40K page visits in total) to quantify detection prevalence and employed techniques. Using custom instrumentation to detect when sites probe for automation, we develop a taxonomy of bot detection techniques and measure how often they appear in practice. Chromium headless encounters a 15% soft block rate compared to 7% for other configurations. Across all conditions, 82% of blocks are attributable to bot detection (59% vendor-confirmed, 23% inferred from condition-dependent blocking), predominantly by providers with integrated bot detection such as Cloudflare (37% block rate) and Akamai (26%). A header spoofing experiment establishes that 75% of Chromium-headless-only blocks are caused by header-level signals alone, yet JavaScript-based environment probing is more extensive than current blocking rates suggest. These findings demonstrate that bot detection creates systematic, provider-correlated sample loss that the web measurement community neither measures nor reports. The downstream effect on specific measurement outcomes remains future work.

Problem

• bot detection
• browser automation
• web measurement
• sample loss
• headless browsers

Innovation

• bot detection
• web measurement
• browser automation
• sample loss
• headless browser