dataset curation

Existing software modeling datasets are often ad hoc constructions lacking rigorous quality assurance, leading to research findings that are difficult to reproduce, compare, and prone to bias. This work proposes the first benchmarking framework specifically designed for model-driven engineering, treating datasets themselves as first-class evaluation targets. By defining clear metrics for quality, representativeness, and task suitability, the framework establishes a unified platform that enables automated analysis of modeling datasets across multiple languages and formats. For the first time, this approach facilitates systematic evaluation of modeling datasets, substantially enhancing the reproducibility, fairness, and scientific rigor of research in the field.

This study systematically evaluates the suitability and effectiveness of synthetic data across three canonical scenarios: data sharing, model training augmentation, and variance reduction in statistical estimation. By integrating formal modeling, theoretical analysis of generative models, and empirical case studies, the work presents the first comprehensive taxonomy of synthetic data applications and delineates their boundaries of applicability. The research elucidates both the potential and fundamental limitations of synthetic data in enhancing privacy preservation, model performance, and statistical stability. It further demonstrates that many existing or proposed use cases are misaligned with the intrinsic properties of synthetic data, thereby providing decision-makers with a principled theoretical framework to assess whether synthetic data is appropriate for addressing specific data availability challenges.

Addressing the challenge of constructing high-quality, domain-specific annotated data—often costly and labor-intensive—this paper proposes a few-shot-driven synthetic data generation paradigm. Given only a small set of user-provided examples, the method retrieves semantically relevant real-world text from large-scale web corpora and leverages instruction-tuned large language models (LLMs) to automatically generate well-formatted, task-specific synthetic training data. It is the first approach to synergistically integrate corpus retrieval with LLM-based augmentation, enabling zero human annotation, domain adaptability, and efficient few-shot generalization. Empirical evaluation across biomedical, medical, and commonsense question answering (QA), as well as summarization tasks, demonstrates that models trained on the generated data achieve a 46-point preference score improvement over human-annotated baselines in summarization, while QA models match or surpass the performance of general-purpose foundation models.

Existing synthetic data evaluation lacks unified, transferable quantitative metrics. This paper proposes a novel evaluation framework grounded in generalized cross-validation (GCV) and domain transfer learning. It constructs a cross-dataset performance matrix and defines two core metrics: *fidelity*, quantifying distributional similarity between synthetic and real data; and *generalization coverage*, measuring the task-transfer capability of synthetic data across diverse real-world source domains. The framework is model-agnostic and enables normalized, comparative evaluation of detectors such as YOLOv5s across heterogeneous datasets—including Virtual KITTI, KITTI, and BDD100K. Experiments demonstrate that the method effectively quantifies synthetic data quality, significantly enhancing evaluation generality, comparability, and utility for model optimization. It establishes a scalable, reproducible, and standardized evaluation paradigm for synthetic data development.

The scarcity of real-world data severely hinders the widespread adoption of subsymbolic AI. To address this challenge, this work proposes a unified reference framework based on digital twins to systematically design and analyze simulation-based synthetic data generation methods for AI training. By integrating digital twin technology, high-fidelity simulation, and synthetic data generation, the framework delineates core components, advantages, and key challenges, offering a methodological foundation for producing high-quality, reproducible training data. This study not only fills the critical gap in the lack of systematic guidance for synthetic data generation but also provides a scalable and reusable technical pathway to mitigate reliance on real-world data.

Existing approaches struggle to effectively quantify the similarity and quality between synthetic and real data in evaluating tool-augmented agents. To address this gap, this work proposes SynAE, a novel framework that establishes the first multi-axis evaluation system tailored for multi-turn tool-use scenarios. SynAE introduces four fine-grained metric categories—assessing task instructions, tool invocations, final outputs, and downstream evaluation performance—to systematically measure synthetic data across dimensions of validity, fidelity, and diversity. Integrating natural language processing, trajectory modeling, and controllable generation techniques, the framework enables a reproducible evaluation pipeline and successfully identifies several representative failure modes in synthetic data generation. Empirical results demonstrate that such multidimensional assessment is essential for enhancing the reliability of agent evaluations.

This study addresses the lack of empirical evaluation regarding whether existing dataset documentation frameworks effectively foster developer reflectivity. Combining mixed-methods thematic analysis with corpus-assisted discourse analysis, the research systematically examines how prevailing documentation frameworks—and their real-world instantiations—cover core dimensions of reflectivity. The findings reveal, for the first time, that current frameworks consistently overlook critical reflective themes. Building on this insight, the authors develop a reflectivity-oriented coding manual and propose an enhanced datasheet template incorporating targeted prompts to elicit deeper reflection. This work offers actionable strategies and practical tools to strengthen the reflective capacity of dataset documentation practices.

This study addresses the challenge of generating high-quality synthetic data for training AI vision systems in agricultural settings, where ready-made CAD models are typically unavailable—especially for highly variable, low-yield objects such as potatoes and stones. The work presents the first systematic comparison of non-CAD approaches, including 3D scanning and image-to-3D reconstruction, to create high-fidelity 3D models of agricultural objects for synthetic data generation. Experimental results demonstrate that object detectors trained on such synthetic data achieve strong performance, and after fine-tuning with only a small amount of real-world data, they approach the accuracy of models trained entirely on real data. These findings validate the feasibility and practical utility of high-fidelity, non-CAD-based 3D modeling for agricultural computer vision tasks.

Accurately estimating model test error under scarce labeled data remains challenging. Method: This paper proposes a novel error estimation paradigm leveraging high-quality synthetic data. Theoretically, we derive a new generalization error upper bound incorporating generator quality constraints, quantifying for the first time how generative model fidelity critically affects estimation bias. Methodologically, we design an interpretable and optimization-friendly synthetic sample construction strategy that jointly leverages generative modeling and generalization theory to enhance assessment reliability. Results: Extensive experiments on both synthetic and real-world tabular datasets demonstrate that our approach consistently outperforms existing baselines, achieving significant and robust improvements in both accuracy and stability of error estimation.

This work addresses the challenge that existing synthetic data generation methods struggle to simultaneously ensure realism, support multiple data formats, enable controlled error injection, and produce corresponding error-free ground-truth data. To this end, we propose GouDa, a novel data generator that unifies support for both relational and non-relational data formats within an extensible framework. GouDa leverages customizable attribute distributions and structured error models to flexibly generate high-fidelity synthetic datasets. Its key innovation lies in the synchronized production of data instances containing controllable errors alongside their error-free counterparts, significantly enhancing the utility of synthetic data in edge-case scenarios and providing robust support for data quality evaluation and machine learning tasks.