simulation study design

The scarcity of real-world data severely hinders the widespread adoption of subsymbolic AI. To address this challenge, this work proposes a unified reference framework based on digital twins to systematically design and analyze simulation-based synthetic data generation methods for AI training. By integrating digital twin technology, high-fidelity simulation, and synthetic data generation, the framework delineates core components, advantages, and key challenges, offering a methodological foundation for producing high-quality, reproducible training data. This study not only fills the critical gap in the lack of systematic guidance for synthetic data generation but also provides a scalable and reusable technical pathway to mitigate reliance on real-world data.

This work addresses the challenges of real-world data scarcity, high acquisition costs, and privacy sensitivity in multimodal AI training by introducing Simula, a novel framework that pioneers inference-driven synthetic data generation without requiring any seed data. By integrating an agent-based architecture with a controllable generation pipeline, Simula enables fine-grained control over data characteristics and computational resource allocation, substantially enhancing the interpretability and scalability of synthetic data. Through a comprehensive multidimensional evaluation protocol, the framework simultaneously validates both the intrinsic quality of the generated data and its effectiveness in downstream tasks across multiple benchmarks, offering a practical pathway and design paradigm for AI development under data-constrained conditions.

Scientific computing artifacts—such as analysis scripts and software prototypes—frequently suffer from poor reproducibility due to environmental heterogeneity, dependency drift, and inadequate documentation, thereby undermining research credibility. To address this, we introduce the first cross-disciplinary, structured, and standardized benchmark dataset for computational experiments, encompassing workflows ranging from single-script executions to multi-language, complex pipelines. Our framework uniformly models metadata, standardizes dependency declarations (e.g., requirements.txt, Dockerfiles), encapsulates multi-language execution procedures, and prescribes a rigorous documentation protocol. The dataset comprises dozens of human-validated, fully reproducible experimental cases, enabling objective, comparable, and reproducible evaluation of reproducibility tools. This work fills a critical gap in the field by providing the first systematic, community-grounded benchmark for assessing computational reproducibility, thereby significantly enhancing the rigor, transparency, and comparability of reproducibility research.

This study addresses the uncontrolled false positive rates and misleading inferences arising from commonly used simulation methods in shift-share designs. We systematically evaluate prevailing inferential approaches in empirical research through a suite of multilevel simulation experiments. By comparing Monte Carlo analysis with counterfactual data-generating mechanisms, we uncover non-monotonic trade-offs among fidelity, sensitivity, and risk of misdirection across simulation designs. We propose a novel “progressive-fidelity simulation framework,” demonstrating that low-fidelity simulations suffice to expose fundamental inferential flaws, whereas high-fidelity simulations detect subtle, previously overlooked biases—substantially improving detection power. The framework balances interpretability and computational efficiency, offering a reproducible and scalable paradigm for assessing the robustness of causal inference methods.

Long-standing deficiencies in standardized, high-quality criteria for data science simulation studies have led to inconsistent design practices, poor reproducibility, and limited external validity. To address this, we propose MERITS—a simulation quality framework for trustworthy data science—systematically defining six core dimensions: Modularity, Efficiency, Realism, Stability, Intuitiveness, and Transparency. MERITS is the first to operationalize the PCS (Predictability-Computability-Stability) theory into concrete design principles and innovatively introduces a “cooking metaphor” to structure simulations as executable “recipes.” The framework includes 13 actionable design guidelines and is validated through empirical reconstruction of existing studies. Designed for cross-disciplinary applicability, MERITS has been successfully applied to diagnostic reconstructions of prior work, yielding substantial improvements in interpretability, reproducibility, and external validity.

This work proposes a GAN-inspired privacy-preserving synthetic data generation method that avoids direct access to original data during training. Instead, it leverages fuzz testing to produce candidate samples and iteratively refines them through a discriminator-guided feedback loop combined with statistical distribution constraints to approximate the original data distribution. By innovatively integrating fuzz testing, adversarial discrimination, and indirect constraint mechanisms, the approach achieves strong privacy guarantees—effectively resisting membership inference and data reconstruction attacks—while preserving high data utility. Extensive experiments on four benchmark datasets demonstrate that the proposed method strikes a superior balance between privacy protection and data fidelity compared to existing techniques.

This work addresses the risks of bias in statistical inference when using synthetic data generated by modern generative AI models—such as diffusion models, GANs, and large language models—due to model misspecification, underestimation of uncertainty, and insufficient generalization. It presents the first systematic integration of generative modeling with statistical inference theory, clarifying the assumptions and conditions under which synthetic data can reliably support scientific discovery. By unifying uncertainty quantification, model diagnostics, and downstream task analysis, the study proposes a principled framework that delineates effective usage guidelines, identifies critical failure modes, and offers practical recommendations for researchers and developers, along with directions for future research.

Existing approaches struggle to effectively quantify the similarity and quality between synthetic and real data in evaluating tool-augmented agents. To address this gap, this work proposes SynAE, a novel framework that establishes the first multi-axis evaluation system tailored for multi-turn tool-use scenarios. SynAE introduces four fine-grained metric categories—assessing task instructions, tool invocations, final outputs, and downstream evaluation performance—to systematically measure synthetic data across dimensions of validity, fidelity, and diversity. Integrating natural language processing, trajectory modeling, and controllable generation techniques, the framework enables a reproducible evaluation pipeline and successfully identifies several representative failure modes in synthetic data generation. Empirical results demonstrate that such multidimensional assessment is essential for enhancing the reliability of agent evaluations.

This work addresses the critical gap in understanding whether synthetic images are truly interchangeable with real ones in model training and the absence of systematic evaluation frameworks to ensure their safe and effective use. The study systematically quantifies discrepancies between synthetic and real images across three dimensions: high-dimensional feature distributions, low-level statistical properties in color space, and model training dynamics. Building on these insights, the authors propose a pre-evaluation metric for synthetic data of unknown quality and a safety-aware data fusion strategy for training. Experiments demonstrate that carefully calibrated mixing ratios and integration methods of synthetic and real data can substantially enhance model performance and robustness, thereby offering both theoretical grounding and practical guidance for the reliable deployment of synthetic data in machine learning pipelines.

This study addresses the limitations of traditional Monte Carlo simulations, which rely on ad hoc assumptions and struggle to generate data reflecting realistic multilevel structures, thereby compromising the validity of quantitative method evaluations. To overcome this, the authors propose the first six-stage workflow integrating generative AI with multilevel data simulation. They innovatively adapt diffusion models and generative adversarial networks (GANs) to accommodate hierarchical data structures and introduce a comprehensive synthetic data quality assessment framework that ensures both within-table and cross-table consistency. Empirical experiments on real-world social science datasets demonstrate that the proposed approach substantially enhances the realism and reliability of Monte Carlo simulations, outperforming conventional strategies and providing a more empirically grounded benchmark for evaluating predictive performance and parameter recovery in quantitative methods.