Beyond One-shot: AI Agents for Learning in Field Experiments

2026-06-01

AI Summary

Traditional A/B testing struggles to automatically distill actionable insights from historical experiments to optimize subsequent interventions. This work proposes a tool-augmented AI agent that, for the first time, integrates the DIKW (Data-Information-Knowledge-Wisdom) reasoning framework with transparent chains of evidence for generating behavioral interventions. Leveraging field experiment data from the domain of medical prescription messaging, the approach employs a two-stage learning and optimization process. Results demonstrate that the AI-generated optimal intervention message achieves a click-through rate of 69.8%, outperforming both general-purpose large language models and theory-driven baselines by 6.5 percentage points. These findings underscore the critical role of domain-specific experimental data in enhancing the efficacy of behavioral interventions.

Abstract

Organizations routinely run experiments for A/B testing, yet the data generated from one experiment is underutilized to inform subsequent intervention design. Significant barriers exist to extracting actionable knowledge from prior experimental data to inform new interventions. We study whether tool-augmented agentic AI can automatically learn from experimental data to generate new interventions in subsequent experiments. Through two-stage field experiments in healthcare prescription messaging (693,139 patient visits), we compare a Human + Chatbot method (Stage 1: behavioral experts with conversational AI co-designing 13 message variants, 444,691 patient visits) against a Tool-Augmented Agentic AI method (Stage 2: AI autonomously extracting principles from Stage 1 data to generate 17 new variants, 248,448 patient visits). The Agentic AI method, equipped with analytical tools, structured Data-Information-Knowledge-Wisdom (DIKW) reasoning agents, and transparent evidence chains, produces superior interventions: the best AI-generated message achieved a 69.8% CTR (+6.5 percentage points over baseline). Critically, our results suggest that the value comes from domain-specific experimental data, not from general reasoning ability: frontier LLMs operating without experimental data failed to predict which interventions would succeed. The field experiments also revealed that general-purpose behavioral theories used for intervention design do not extend uniformly to specific healthcare contexts, motivating an agentic AI approach to theory audits at field-experiment scale. Our research shows that tool-augmented AI can learn from experimental data and generate improved domain-relevant interventions, transforming behavioral experimentation from one-shot evaluation into a scalable system for cumulative design learning.

Problem

• field experiments
• intervention design
• experimental data reuse
• behavioral interventions
• cumulative learning

Innovation

• Tool-Augmented Agentic AI
• Field Experiment Learning
• DIKW Reasoning
• Behavioral Intervention Design
• Cumulative Experimentation