Can AI Agents Simulate A/B Test Outcomes? A Validation Framework for Agentic Experimentation

2026-08-03

AI Summary

A/B testing is costly, necessitating effective pre-screening approaches. This work proposes a Simulated Randomized Controlled Trial (S-RCT) framework that leverages AI agents to simulate experimental outcomes based on user profiles and intervention descriptions. It introduces a novel two-stage pre-experiment calibration protocol and a within-subject design, alongside a two-level error decomposition that disentangles agent approximation error from sampling error. The framework is compatible with arbitrary behavioral models and requires no agent customization. Evaluation across 67 real-world marketing A/B tests demonstrates that post-calibration predictions reduce mean squared error by approximately 77-fold, the within-subject design lowers standard errors by 2.4-fold, and effect direction consistency reaches 0.70, substantially enhancing simulation accuracy and practical utility.

Abstract

A/B testing remains the standard for rolling out new features in the technology industry. Each experiment, however, consumes real traffic, engineering effort, and weeks of wall-clock time. Can AI agents---conditioned on behavioral profiles and contextual descriptions of the intervention---simulate outcomes accurately enough to vet candidate treatments before committing live traffic? We formalize this question as a \emph{Simulated Randomized Controlled Trial} (S-RCT) and derive a two-layer error decomposition that separates agent approximation error from subsampling error, enabling targeted improvements to each. The framework is agent-agnostic: any behavioral model---from a fine-tuned specialist to a general-purpose foundation model---can serve as the simulation engine. Validated on 67 historical marketing A/B tests, a baseline S-RCT using an off-the-shelf foundation model captures directional signal (sign overlap 0.70) but systematically overshoots effect magnitudes. A two-phase pre-period calibration protocol reduces the squared prediction error (after removing irreducible measurement noise) by ${\sim}77\times$; a within-subject design---where each agent is exposed to both arms---reduces standard errors by ${\sim}2.4\times$. We discuss limitations of the current approach and identify applications where experimenters stand to benefit from agentic signals.

Problem

• A/B testing
• AI agents
• simulation
• randomized controlled trial
• experimentation

Innovation

• Simulated Randomized Controlled Trial
• AI Agents
• A/B Testing
• Error Decomposition
• Pre-period Calibration