Valid Inference with Synthetic Data via Task Exchangeability

2026-06-11

AI Summary

This work addresses the challenge that synthetic data—often plagued by biases, noise, or model misspecification—struggles to support reliable scientific inference. To overcome this limitation, the authors propose a statistical framework grounded in task exchangeability, which leverages exchangeable relationships between historical real-world tasks and the current target task. This framework provides, for the first time, theoretical guarantees for valid inference under synthetic data and extends naturally to settings where strict exchangeability does not hold. By integrating task modeling, calibration techniques, and uncertainty quantification, the approach yields well-calibrated inferential results with proper coverage, as demonstrated in applications to public opinion polling and AI-based automated scoring.

Abstract

There is a proliferation of work arguing for the use of synthetic data in scientific research. For example, social scientists are arguing for the use of LLM-generated "silicon samples" in pilot studies; AI evaluations increasingly rely on "LLM-as-a-judge" outputs; and proteomics research is accelerated by generative models that produce synthetic protein structures. These developments raise an intriguing possibility: synthetic data may help researchers ask more questions, run more studies, and accelerate discovery. But they also raise a fundamental concern: synthetic data can be biased, noisy, and misspecified. In this work, we propose statistical principles for using synthetic data in scientific research with provable validity guarantees. The key insight is a new technical condition that we call task exchangeability. Informally, this is a requirement that the researcher can identify historical tasks, for which real data is available, such that their current task of interest is exchangeable with the historical tasks in an appropriate mathematical sense. We develop methods for valid inference under task exchangeability, together with extensions that provide guarantees even beyond exchangeability. We demonstrate the framework on public opinion surveys with silicon samples and AI evaluation with autoraters.

Problem

• synthetic data
• valid inference
• task exchangeability
• statistical validity
• bias

Innovation

• task exchangeability
• synthetic data
• valid inference
• statistical guarantees
• LLM-generated data