🤖 AI Summary
Existing agent monitoring evaluation methods struggle to detect highly stealthy and diverse attacks, leading to an overestimation of their defensive capabilities. This work proposes a semi-automated red-teaming framework that systematically constructs diverse attack trajectories through a three-stage pipeline: strategy generation, execution, and trajectory optimization. The approach innovatively introduces an attack taxonomy to mitigate mode collapse, decomposes the attack construction process to bridge the gap between conception and execution, and leverages large language models to enable scalable, semi-automated testing. Using this framework, the authors develop MonitoringBench—a benchmark comprising 2,644 attack trajectories within BashArena—which reduces the detection rate of state-of-the-art monitors from 94.9% to 60.3%, exposing critical weaknesses in their defenses against persuasive attacks and in the calibration of risk scoring mechanisms.
📝 Abstract
We introduce a red-teaming methodology that exposes harder-to-catch attacks for coding-agent monitors, suggesting that current practices may under-elicit attacks and overstate monitor performance. We identify three challenges with current red-teaming. First, mode collapse in attack generation, which we reduce with a novel attack taxonomy for broader coverage. Second, a conceive-execute gap: frontier LLMs can propose strong attack ideas or execute them, but not all at once. We mitigate this by decomposing attack construction into strategy generation, execution, and post-hoc trajectory refinement. Third, manual elicitation is costly to scale, which we address with our semi-automated red-teaming pipeline. Applied to BashArena, an AI control setting for tool-using coding agents, this pipeline produces MonitoringBench, a benchmark of 2,644 attack trajectories for evaluating monitor capabilities and failure modes. Our pipeline produces more diverse and stronger attacks: Opus-4.5 monitor's catch rate falls from 94.9\% on elicited-only Opus attacks to 60.3\% on our best refined attacks, with larger drops for several mid-tier monitors. Attacks optimized against three development monitors generalize to ten held-out monitors, with catch rates generally increasing with monitor capability. Using this benchmark, we provide a snapshot of the current monitor capabilities and find that frontier monitors often detect suspicious actions but fall for persuasion or fail to calibrate suspiciousness scores appropriately, suggesting tractable paths for improvement. MonitoringBench provides both a static benchmark for current tool-use monitors and a reusable methodology for refreshing these evaluations as agents and monitors improve.