A Human-Augmenting Agentic Workflow for Observational Causal Inference

2026-07-24

AI Summary

This work addresses the pressing need for automation in observational causal inference to alleviate the burden of repetitive tasks on domain experts. We propose oci-agent, the first systematic human-in-the-loop agent framework that automatically performs covariate balance checks, propensity score trimming, and sensitivity analyses across diverse settings—including single binary treatments, heterogeneous treatment effects, and multiple continuous treatments. Integrating doubly robust estimation with partially linear models, oci-agent features automated diagnostics and interactive feedback mechanisms, and is accompanied by an open-source Python package. Evaluated on both internal Netflix data and public benchmarks, the framework runs over one hundred times per month in production, significantly outperforming unstructured baselines and matching the performance of meticulously hand-tuned expert analyses, thereby establishing a new paradigm for human–AI collaboration in causal inference.

Abstract

Data analysis agents are becoming increasingly common tools for applied and scientific research. Yet, for highly specialized tasks such as Observational Causal Inference (OCI), human oversight remains necessary to ensure the validity of results. We introduce `oci-agent`, an open-source Python package that implements a human-in-the-loop agentic workflow for observational causal inference. `oci-agent` is designed to automate vital but laborious aspects of applied causal inference, such as covariate balance checking, propensity score trimming, and sensitivity analysis, so that humans can focus on more nuanced tasks, such as framing questions, scrutinizing assumptions, and evaluating diagnostics and results. We initially open-sourced `oci-agent` in June 2026 with support for doubly robust learning of the average treatment effect of a single binary treatment. Since then, we have added support for heterogeneous treatment effect estimation and for multiple continuous treatments via partially linear models. In this paper, we describe the principles behind `oci-agent` and offer internal Netflix case studies and evaluations on public data of its capabilities. Across numerous evaluations, `oci-agent` outperforms less structured baselines while remaining competitive with hand-tuned benchmarks. `oci-agent` is used extensively for causal inference at Netflix and has orchestrated more than 100 analyses per month since its release in June.

Problem

• Observational Causal Inference
• Human-in-the-loop
• Causal Analysis
• Automated Data Analysis
• Treatment Effect Estimation

Innovation

• human-in-the-loop
• observational causal inference
• agentic workflow
• automated causal analysis
• heterogeneous treatment effects