Statistical Inference Leveraging Synthetic Data with Distribution-Free Guarantees

2025-09-24

AI Summary

This work addresses the challenge of safely and controllably improving statistical inference sample efficiency using high-quality synthetic data—such as AI-generated or cross-task transferred data—without assuming knowledge of the synthetic data’s underlying distribution. To this end, we propose the General Synthetic-Powered Inference (GESPI) framework, which adaptively weights real and synthetic data to seamlessly integrate mainstream inference methodologies—including conformal prediction, risk-controlled estimation, hypothesis testing, and multiple testing—while preserving the original inference pipeline. Crucially, GESPI automatically reverts to real-data-only inference when synthetic data quality is poor, guaranteeing that inference error remains within user-specified bounds and monotonically decreases as synthetic quality improves. This constitutes the first distribution-agnostic framework for safe inference enhancement. Empirically, GESPI significantly boosts statistical power in low-label tasks, including AlphaFold-based protein structure prediction and large-language-model-driven mathematical reasoning.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Probabilistic Inference
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Agentic search

Abstract

The rapid proliferation of high-quality synthetic data -- generated by advanced AI models or collected as auxiliary data from related tasks -- presents both opportunities and challenges for statistical inference. This paper introduces a GEneral Synthetic-Powered Inference (GESPI) framework that wraps around any statistical inference procedure to safely enhance sample efficiency by combining synthetic and real data. Our framework leverages high-quality synthetic data to boost statistical power, yet adaptively defaults to the standard inference method using only real data when synthetic data is of low quality. The error of our method remains below a user-specified bound without any distributional assumptions on the synthetic data, and decreases as the quality of the synthetic data improves. This flexibility enables seamless integration with conformal prediction, risk control, hypothesis testing, and multiple testing procedures, all without modifying the base inference method. We demonstrate the benefits of our method on challenging tasks with limited labeled data, including AlphaFold protein structure prediction, and comparing large reasoning models on complex math problems.

Problem

• Enhancing statistical inference by combining synthetic and real data
• Providing distribution-free guarantees for error control in inference
• Improving sample efficiency in tasks with limited labeled data

Innovation

• Combines synthetic and real data for efficiency
• Adaptively defaults to real data when needed
• Maintains error guarantees without distributional assumptions