Remote Auditing: Design-based Tests of Randomization, Selection, and Missingness with Broadly Accessible Satellite Imagery

2025-09-30

AI Summary

This paper addresses the challenge of verifying the independence of intervention assignment in randomized controlled trials (RCTs) without baseline surveys or parametric assumptions. It proposes a remote audit method leveraging pre-treatment satellite imagery, implementing a conditional randomization test combined with pre-registered study design and a max-statistic correction across multiple models and spatial resolutions to rigorously control for multiple comparisons—applicable to both block and cluster-randomized designs. The key contribution is the first finite-sample, nonparametric, pre-registered remote-sensing–driven audit framework, substantially enhancing the robustness and reproducibility of conventional balance tests. Empirically, the method confirms randomization validity and detects selective attrition risk in Uganda’s Youth Opportunities Project; in a Bangladeshi school trial, it reveals that intervention assignment is significantly predictable from satellite-derived features, exposing potential implementation bias.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Stochastic Optimization
• Intelligent Robots: State Estimation

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Randomized controlled trials (RCTs) are the benchmark for causal inference, yet field implementation can deviate. We here present a remote audit - a design-based, preregistrable diagnostic that uses only pre-treatment satellite imagery to test whether assignment is independent of local conditions. The conditional randomization test of the remote audit evaluates whether treatment assignment is more predictable from pre-treatment satellite features than expected under the experiment's registered mechanism, providing a finite-sample valid, design-based diagnostic that requires no parametric assumptions. The procedure is finite-sample valid, honors blocks and clusters, and controls multiplicity across image models and resolutions via a max-statistic. We illustrate with two RCTs: Uganda's Youth Opportunities Program, where the audit corroborates randomization and flags selection and missing-data risks; and a school-based trial in Bangladesh, where assignment is highly predictable from pre-treatment features relative to the stated design, consistent with independent concerns about irregularities. Remote audits complement balance tests, lower early-stage costs, and enable rapid design checks when baseline surveys are expensive or infeasible.

Problem

• Tests RCT randomization integrity using pre-treatment satellite imagery
• Detects selection bias and missing data risks in field experiments
• Provides design-based diagnostics without parametric assumptions or baseline surveys

Innovation

• Uses pre-treatment satellite imagery for randomization testing
• Applies finite-sample valid conditional randomization tests
• Controls multiplicity via max-statistic across image resolutions