Small Samples and Short Panels: Evaluating Policy Evaluation Methods with Realistic Data

2026-08-02

AI Summary

This study addresses the challenges of causal inference in quasi-experimental settings—such as health policy research—where small samples and short panels often lead to substantial bias and undercoverage of confidence intervals. To bridge the gap between theoretical assumptions and empirical practice, the authors innovatively integrate calibrated simulations with real-data reconstruction to systematically evaluate, for the first time, the finite-sample performance of Synthetic Difference-in-Differences (SDiD) and Augmented Synthetic Control Method (ASCM) under realistic small-sample, short-panel conditions. The analysis delineates the reliability boundaries of these two approaches across varying data configurations, thereby filling a critical empirical void left by the absence of formal theoretical guarantees. The findings offer evidence-based guidance for researchers and policymakers in selecting appropriate methods for causal inference in data-constrained environments.

Abstract

Methods for estimating causal effects in longitudinal, quasi-experimental settings are widely used in economics, public health, political science, and other fields. However, studies evaluating effects of health policies often rely on limited sample sizes, both in terms of study units and time periods analyzed. Synthetic difference-in-differences (SDiD) and the augmented synthetic control method (ASCM) are recently developed methods for evaluating the effects of policy interventions. Although SDiD and ASCM generally rely on weaker assumptions than both DiD and SCM, they lack theoretical performance guarantees with respect to bias or coverage in relevant settings with small sample sizes or short panel lengths. To evaluate the performance of SDiD and ASCM in realistic, small-sample settings, we employ a calibrated simulation strategy that allows the injection of a known treatment effect into existing data in a setting of interest. Drawing on findings from these empirical investigations, we offer practical guidance for researchers and policymakers on when SDiD and ASCM are likely to yield reliable estimates and inferences under realistic scenarios.

Problem

• small samples
• short panels
• policy evaluation
• causal inference
• synthetic control

Innovation

• synthetic difference-in-differences
• augmented synthetic control method
• small samples
• calibrated simulation
• causal inference