synthetic data generation

The scarcity of real-world data severely hinders the widespread adoption of subsymbolic AI. To address this challenge, this work proposes a unified reference framework based on digital twins to systematically design and analyze simulation-based synthetic data generation methods for AI training. By integrating digital twin technology, high-fidelity simulation, and synthetic data generation, the framework delineates core components, advantages, and key challenges, offering a methodological foundation for producing high-quality, reproducible training data. This study not only fills the critical gap in the lack of systematic guidance for synthetic data generation but also provides a scalable and reusable technical pathway to mitigate reliance on real-world data.

This study systematically evaluates the suitability and effectiveness of synthetic data across three canonical scenarios: data sharing, model training augmentation, and variance reduction in statistical estimation. By integrating formal modeling, theoretical analysis of generative models, and empirical case studies, the work presents the first comprehensive taxonomy of synthetic data applications and delineates their boundaries of applicability. The research elucidates both the potential and fundamental limitations of synthetic data in enhancing privacy preservation, model performance, and statistical stability. It further demonstrates that many existing or proposed use cases are misaligned with the intrinsic properties of synthetic data, thereby providing decision-makers with a principled theoretical framework to assess whether synthetic data is appropriate for addressing specific data availability challenges.

This work addresses the critical gap in understanding whether synthetic images are truly interchangeable with real ones in model training and the absence of systematic evaluation frameworks to ensure their safe and effective use. The study systematically quantifies discrepancies between synthetic and real images across three dimensions: high-dimensional feature distributions, low-level statistical properties in color space, and model training dynamics. Building on these insights, the authors propose a pre-evaluation metric for synthetic data of unknown quality and a safety-aware data fusion strategy for training. Experiments demonstrate that carefully calibrated mixing ratios and integration methods of synthetic and real data can substantially enhance model performance and robustness, thereby offering both theoretical grounding and practical guidance for the reliable deployment of synthetic data in machine learning pipelines.

This work addresses the challenges of real-world data scarcity, high acquisition costs, and privacy sensitivity in multimodal AI training by introducing Simula, a novel framework that pioneers inference-driven synthetic data generation without requiring any seed data. By integrating an agent-based architecture with a controllable generation pipeline, Simula enables fine-grained control over data characteristics and computational resource allocation, substantially enhancing the interpretability and scalability of synthetic data. Through a comprehensive multidimensional evaluation protocol, the framework simultaneously validates both the intrinsic quality of the generated data and its effectiveness in downstream tasks across multiple benchmarks, offering a practical pathway and design paradigm for AI development under data-constrained conditions.

To address the pervasive diminishing returns in synthetic data augmentation, this paper proposes DP, a dynamic synthetic data generation framework inspired by the pedagogical principle of “deliberate practice.” Departing from conventional “generate-then-prune” paradigms, DP directly targets high-value sample distributions via three core mechanisms: dynamic difficulty adjustment, information-theoretic sample generation, and theory-guided sample selection. It is the first work to formally incorporate human learning principles—specifically, challenge-based training—into synthetic data generation and theoretically proves its positive impact on model scaling laws. Implemented as a lightweight plugin, DP seamlessly integrates with mainstream diffusion models and large language models. Experiments demonstrate significant efficiency gains: on ImageNet-100, it reduces required sample count and training iterations by 3.4× and 6×, respectively; on ImageNet-1K, reductions reach 8× in samples and 30% in iterations—while consistently outperforming state-of-the-art methods across all benchmarks.

This work addresses the lack of systematic evaluation for Vision-Language-Action (VLA) models in sim-to-real transfer for dexterous manipulation tasks by introducing a real-world evaluation protocol that encompasses variations in background, lighting, distractors, object types, and spatial configurations. Through over 10,000 physical trials, the study systematically investigates the impact of multi-level domain randomization, photorealistic rendering, high-fidelity physics modeling, and reinforcement learning strategies on generalization performance. It presents the first comprehensive assessment of VLA models’ sim-to-real capabilities in real-world dexterous manipulation, establishes a standardized benchmark, and releases an open-source robotic platform, thereby significantly enhancing the robustness and reproducibility of policies in complex real-world scenarios.

This work addresses the risks of bias in statistical inference when using synthetic data generated by modern generative AI models—such as diffusion models, GANs, and large language models—due to model misspecification, underestimation of uncertainty, and insufficient generalization. It presents the first systematic integration of generative modeling with statistical inference theory, clarifying the assumptions and conditions under which synthetic data can reliably support scientific discovery. By unifying uncertainty quantification, model diagnostics, and downstream task analysis, the study proposes a principled framework that delineates effective usage guidelines, identifies critical failure modes, and offers practical recommendations for researchers and developers, along with directions for future research.

Existing visual simulation platforms suffer from high technical barriers and lack user-friendly, controllable, and interactive environments for non-graphics experts, hindering efficient synthetic data generation, out-of-distribution (OOD) evaluation, and closed-loop agent testing. To address this, we propose LychSim—the first Unreal Engine 5-based simulation framework natively integrating the Model Context Protocol (MCP). LychSim enables language-driven dynamic scene editing and precise pose control through a lightweight Python API, procedural high-fidelity scene generation, and semantically aligned 3D annotations. This framework substantially lowers the entry barrier and has been successfully applied to synthetic data engines, adversarial evaluation in reinforcement learning, and language-guided layout generation. The code and annotated datasets will be open-sourced to foster community advancement.

This study investigates the fundamental limitations of synthetic data augmentation in enhancing sample information for statistical inference, with particular emphasis on its theoretical constraints when incorporating prior knowledge. Treating synthetic data as a model of the prior, we formally define the synthetic distribution within both maximum likelihood and Bayesian frameworks. By integrating Fisher information, information theory, and statistical decision theory, we establish an intrinsic upper bound on the marginal information gain achievable through synthetic data. Our analysis reveals that naive prior specifications lack epistemic justification and generally fail to improve conventional inferential performance. Nevertheless, under the training/test partition paradigm, synthetic data can effectively regularize high-dimensional model spaces by imposing structural constraints, thereby serving a beneficial regularization role despite its limited informational contribution.

Controllable human video generation is hindered by the scarcity of real-world data, particularly for rare identities and complex motion scenarios. This work proposes a unified diffusion-based framework that systematically investigates, for the first time, the synergistic mechanisms between synthetic and real data in human-centric video generation. It reveals their complementary roles and introduces an efficient synthetic sample selection strategy to enhance training. The proposed approach significantly improves motion realism, temporal coherence, and identity fidelity in generated videos, establishing a new paradigm for building data-efficient and generalizable controllable video generation models.