CIAware-Bench: Benchmarking Control Intervention Awareness Across Frontier LLMs

2026-06-09

AI Summary

This work addresses the emerging risk that state-of-the-art large language models (LLMs) may detect and circumvent external control interventions—such as trajectory modifications—thereby undermining AI safety mechanisms. The study introduces the first systematic definition and quantification of “control intervention awareness” (CI-awareness), along with CIAware-Bench, a benchmark spanning four domains: argumentative writing, BigCodeBench, Bash Arena, and SHADE-Arena. Leveraging trajectory watermarking, auxiliary tasks, and diverse control protocols, the authors evaluate CI-awareness across 11 leading models via binary classification accuracy. Results reveal generally low to moderate CI-awareness (peak accuracy 0.87 versus random baseline 0.5), with higher detectability across model families. These findings indicate that CI-awareness is not an intrinsic property but rather contingent on task domain, model lineage, and deployment context, highlighting the presence of vendor-specific behavioral signatures.

Abstract

AI control protocols oversee untrusted models by monitoring their actions and modifying potentially unsafe steps, often using a trusted model. This partially tampers with the untrusted model's trajectory. If the trusted model detects such an intervention, it may infer properties of the monitor and adapt to evade control. We introduce \textbf{CIAware-Bench}, a benchmark for measuring \textbf{c}ontrol \textbf{i}ntervention (CI) awareness across frontier models. CIAware-Bench tests whether models can distinguish their own trajectories from those modified by a control intervention. The benchmark is comprised of a suite of four task domains (essay writing, BigCodeBench, Bash Arena, and SHADE-Arena), while varying trajectory watermarking, side-task presence, and the control protocol. Evaluating eleven frontier models, we find low to moderate CI awareness under default settings (up to 0.87; random chance balanced binary classification accuracy is 0.5) with substantial variation across task domains and model pairs. Detection is generally easier across model families, suggesting that models exploit provider-specific differences in style or post-training. Overall, CI awareness is not a fixed model-level property, and should be measured for each new model release and deployment scenario. We release CIAware-Bench to track CI awareness and inform control protocols whose interventions are harder to detect.

Problem

• control intervention
• intervention awareness
• LLMs
• AI control
• trajectory tampering

Innovation

• Control Intervention Awareness
• CIAware-Bench
• AI Safety
• Model Monitoring
• Trajectory Tampering