data collection protocols

This work addresses the frequent neglect of sampling strategy design and generalizability in software engineering research, which often undermines the representativeness of empirical findings. To remedy this, the paper introduces a domain-specific language (DSL) that explicitly models complex sampling workflows over code repositories through composable sampling operators, enabling—for the first time—formal specification and reasoning about the generalizability of sampling strategies. Implemented as a fluent Python API, the DSL is integrated with a statistical metric system to quantitatively assess the external validity of sampled datasets. The authors demonstrate the expressiveness and practical utility of their approach by reconstructing and formalizing the sampling procedures from multiple Mining Software Repositories (MSR) studies, thereby validating the framework’s capacity to capture real-world methodological diversity.

This paper addresses performance degradation of machine learning models in specific deployment environments (e.g., a hospital or national park) due to distributional shift. We formalize the Deployment-Specialized Subset Selection (DS3) task: selecting an optimal subset from a general training set to maximize model performance in the target environment. We introduce DataS³, the first cross-domain, multi-scenario benchmark for DS3, and empirically demonstrate the systematic failure of mainstream data selection methods on this task. To overcome these limitations, we propose a novel DS3 framework integrating coresets, distribution-aware filtering, and human curation—enabling subset optimization even without labeled deployment data. Experiments show that expert-curated subsets yield average accuracy gains of 20.7%, with peaks up to 51.3%, significantly outperforming full-dataset training and existing selection baselines. Moreover, DS3 improves training efficiency and enhances generalization robustness across diverse domains.

Existing software modeling datasets are often ad hoc constructions lacking rigorous quality assurance, leading to research findings that are difficult to reproduce, compare, and prone to bias. This work proposes the first benchmarking framework specifically designed for model-driven engineering, treating datasets themselves as first-class evaluation targets. By defining clear metrics for quality, representativeness, and task suitability, the framework establishes a unified platform that enables automated analysis of modeling datasets across multiple languages and formats. For the first time, this approach facilitates systematic evaluation of modeling datasets, substantially enhancing the reproducibility, fairness, and scientific rigor of research in the field.

Existing synthetic data evaluation lacks unified, transferable quantitative metrics. This paper proposes a novel evaluation framework grounded in generalized cross-validation (GCV) and domain transfer learning. It constructs a cross-dataset performance matrix and defines two core metrics: *fidelity*, quantifying distributional similarity between synthetic and real data; and *generalization coverage*, measuring the task-transfer capability of synthetic data across diverse real-world source domains. The framework is model-agnostic and enables normalized, comparative evaluation of detectors such as YOLOv5s across heterogeneous datasets—including Virtual KITTI, KITTI, and BDD100K. Experiments demonstrate that the method effectively quantifies synthetic data quality, significantly enhancing evaluation generality, comparability, and utility for model optimization. It establishes a scalable, reproducible, and standardized evaluation paradigm for synthetic data development.

Dataset quality defects—such as missing documentation, incorrect labels, and ethical risks—are pervasive in open platforms yet resistant to detection by rule-based scripts, necessitating intelligent, automated identification methods. Method: We introduce the first LLM-agent benchmark for discovering real-world dataset quality issues, covering 221 empirically validated cases across eight platforms. It uniquely evaluates agents’ ability to autonomously detect latent defects without prior prompting. We propose an automated evaluation framework powered by GPT-4o, achieving high agreement with human experts (Cohen’s κ = 0.89), and ensure benchmark reliability via multi-source real-data sampling and expert annotation. Contribution/Results: Experiments reveal that even the state-of-the-art Curator agent detects only ~30% of defects, underscoring task difficulty. All benchmark data, code, and evaluation tools are publicly released to advance intelligent data governance.

This work addresses the challenge that domain experts face in translating natural language descriptions of data quality requirements into executable analyses, a process often hindered by reliance on data engineers, resulting in inefficiency and high technical barriers. To overcome this, the paper proposes a no-code, model-driven pipeline that leverages a QPM metamodel to define domain-specific quality analysis templates. Coupled with the Constrainify toolchain, it automatically transforms natural language requirements into executable and reusable analytical logic. By integrating model-driven engineering, metamodeling, and no-code web technologies, the approach significantly reduces dependency on technical expertise, enabling efficient, reproducible, and semantically aligned data quality assessments. This advancement enhances both the accessibility and automation of data quality analysis for non-technical domain practitioners.

Existing approaches struggle to effectively quantify the similarity and quality between synthetic and real data in evaluating tool-augmented agents. To address this gap, this work proposes SynAE, a novel framework that establishes the first multi-axis evaluation system tailored for multi-turn tool-use scenarios. SynAE introduces four fine-grained metric categories—assessing task instructions, tool invocations, final outputs, and downstream evaluation performance—to systematically measure synthetic data across dimensions of validity, fidelity, and diversity. Integrating natural language processing, trajectory modeling, and controllable generation techniques, the framework enables a reproducible evaluation pipeline and successfully identifies several representative failure modes in synthetic data generation. Empirical results demonstrate that such multidimensional assessment is essential for enhancing the reliability of agent evaluations.

This work addresses the limited reproducibility of behavioral validation in robotic simulation testing, which often stems from insufficiently documented test configurations, execution protocols, and post-processing procedures. To overcome this, the study proposes a deep integration of data provenance and FAIR (Findable, Accessible, Interoperable, Reusable) principles throughout the entire test generation pipeline—rather than merely appending them to final datasets. The authors extend an existing simulation testing framework by embedding machine-readable, structured metadata at every stage, thereby enabling end-to-end traceable validation workflows. This approach significantly enhances the reproducibility of mobile robot navigation datasets. Additionally, the project distills practical FAIR implementation guidelines tailored to robotics, identifying key challenges such as vocabulary alignment, attribute selection, and adoption of community standards, and offers actionable recommendations for addressing them.

This study addresses the challenge that domain experts, due to limited query language proficiency, often struggle to independently conduct context-specific data quality analyses and must rely on technical specialists, resulting in inefficient workflows. To overcome this limitation, the paper proposes the Quality Pattern Model (QPM) framework—a novel, template-based mechanism that is agnostic to both database technologies and application domains, enabling non-technical users to autonomously define data quality analysis logic. Leveraging a model-driven approach, the authors implement QPM prototypes over XML, RDF, and Neo4j. Experimental results demonstrate that QPM’s expressiveness matches or exceeds that of mainstream query languages while significantly enhancing domain experts’ analytical autonomy. The framework’s effectiveness has been validated in the cultural heritage domain.

This study addresses the compliance challenges faced by data practitioners in machine learning systems under regulations such as the GDPR and the AI Act, particularly concerning data quality. Through semi-structured interviews with practitioners in the European Union, combined with thematic analysis of regulatory texts and engineering workflows, the research systematically uncovers a structural disconnect between regulation-driven data quality requirements and ML engineering practices. It identifies five core challenges: misalignment between legal principles and engineering implementation, fragmented data pipelines, lack of purpose-built compliance tools, ambiguous accountability, and reactive responses to audits. Building on these findings, the work proposes directions for designing compliance-oriented tooling, establishing effective governance mechanisms, and fostering cultural transformation to bridge the gap between regulatory mandates and practical ML development.