dataset annotation

This study addresses critical challenges in managing data labeling requirements for autonomous driving AI perception systems (AIePS), including ambiguity, edge-case complexity, dynamic evolution, inconsistency, and resource constraints. Through semi-structured interviews and thematic analysis with 19 practitioners from six international industry organizations and four research institutions, we empirically investigate how labeling requirement definition, quality assurance, and evolution impact system safety, reliability, and regulatory compliance. We present the first empirical characterization of the propagation pathway linking labeling requirements to AI system performance. Based on our findings, we propose a novel three-dimensional best-practice framework integrating ethical and regulatory compliance, labeling guideline optimization, and embedded quality assurance. This work advances the interdisciplinary frontier of Software Engineering for AI (SE4AI) and Requirements Engineering for AI (RE4AI), offering actionable, evidence-based strategies to improve labeling quality, enhance system robustness, and support regulatory adherence.

This study addresses the inconsistency in human annotation caused by ambiguous category definitions in traditional content moderation. To resolve this, the authors propose an AI-driven constitutional annotation framework: large language models first assist humans in formulating structured, interpretable category “constitutions,” which then guide automated dual-axis labeling of intent and content safety. This approach shifts human effort from case-by-case judgments to high-level semantic definition. Evaluated on harassment, hate speech, and non-violent criminal conduct tasks, the method reduces cross-model annotation inconsistency by up to 57-fold compared to conventional paragraph-based rules and effectively exposes latent gaps in existing policy formulations.

Scientific image annotation projects face cross-domain managerial challenges—including scarce data acquisition, inefficient resource allocation, inadequate annotator training, and pronounced human bias. To address these issues, this paper proposes the first general-purpose framework for preparing scientific image annotation projects. The framework systematically integrates objective definition, data availability assessment, multi-role team configuration, bias mitigation strategies, and an iterative annotator training mechanism, complemented by a recommended toolchain supporting integrated project management, quality control, and collaborative annotation. A novel closed-loop workflow—comprising bias detection, feedback integration, and retraining—is introduced to significantly enhance annotation consistency and efficiency. Empirical evaluation across multiple disciplines demonstrates that the framework reduces annotation costs by over 20%, improves project success rates, and strengthens knowledge base construction quality—thereby filling a critical research gap in standardized preparation guidelines for complex scientific image annotation.

This study addresses a critical gap in machine learning education: the overreliance on pre-labeled datasets, which often obscures the subjectivity and ambiguity inherent in data annotation, leading students to place undue trust in model outputs. To counter this, the authors introduce an innovative pedagogical intervention that transforms manual annotation into an active learning tool. Students annotated hair coverage in skin lesion images using a three-point scale, followed by structured reflections via questionnaires. A cross-institutional experiment involving 43 participants from Fontys University of Applied Sciences (Netherlands) and the IT University of Copenhagen (Denmark) demonstrated that this approach significantly enhanced learners’ awareness of annotation ambiguity, dataset biases, and model limitations. Most participants acknowledged the influence of personal interpretation on labeling decisions and reported higher engagement compared to traditional instruction. This work provides the first empirical evidence supporting subjective annotation as an effective strategy for cultivating critical thinking about AI systems.

This study addresses the widespread neglect in machine learning research of when validation occurs during data annotation—a critical factor influencing both label quality and cost—despite overreliance on post-hoc quality control. Drawing inspiration from the “shift-left” principle in software engineering, this work proposes a tripartite classification of quality checkpoints across early, intermediate, and late stages of the annotation pipeline and introduces a parameterized error propagation model that, for the first time, treats validation timing as a quantifiable design variable. Through error propagation modeling, process decomposition, and literature analysis, the authors find that only 4% of recent studies report validation timing. Their analysis demonstrates that early-stage quality checks can reduce error correction costs by up to two orders of magnitude. The paper calls for standardized reporting of timing configurations, platform support for tunable timing parameters, and empirical studies on stage-specific detection rates.

Manual annotation suffers from inconsistent quality and lacks systematic evaluation. Method: This paper proposes a consensus-based quality measurement framework grounded in multi-round annotation statistics, using dynamic decay of inter-annotator agreement variance as the core metric—established here as a “gold standard” for data quality. Recognizing annotators’ significant warm-up period but prohibitive cost of full-sample multiple annotation, we design a low-redundancy, high-efficiency progressive annotation protocol. The approach integrates statistical consistency analysis, variance convergence modeling, and label confidence estimation. Contribution/Results: Our paradigm substantially enhances data quality’s measurability and controllability: experiments show 3.2–7.8% accuracy gains across multiple NLP tasks and over 30% reduction in annotation redundancy.

This study addresses the compliance challenges faced by data practitioners in machine learning systems under regulations such as the GDPR and the AI Act, particularly concerning data quality. Through semi-structured interviews with practitioners in the European Union, combined with thematic analysis of regulatory texts and engineering workflows, the research systematically uncovers a structural disconnect between regulation-driven data quality requirements and ML engineering practices. It identifies five core challenges: misalignment between legal principles and engineering implementation, fragmented data pipelines, lack of purpose-built compliance tools, ambiguous accountability, and reactive responses to audits. Building on these findings, the work proposes directions for designing compliance-oriented tooling, establishing effective governance mechanisms, and fostering cultural transformation to bridge the gap between regulatory mandates and practical ML development.

This study addresses the lack of systematic approaches for constructing, storing, and sharing high-quality annotated corpora. It proposes a generalizable and reusable end-to-end methodology encompassing annotation guideline development, corpus annotation, data storage, sharing mechanisms, and value realization, with an emphasis on full lifecycle management and cross-domain applicability. Integrating linguistic annotation theory, data management standards, and collaborative research practices, the approach is articulated through a structured framework and illustrative examples to yield a clear and actionable guide. The resulting methodology provides standardized support for diverse research domains, significantly enhancing the efficiency and quality with which researchers can build and utilize annotated textual data.

This study addresses the compliance challenges faced by machine learning systems in the European Union, where data quality practices often misalign with regulatory requirements due to a lack of actionable guidance. It presents the first systematic mapping of data quality dimensions to specific provisions of EU regulations, resulting in a practical compliance framework. Through an online survey of over 180 European practitioners, combined with regulatory analysis and empirical investigation, the research uncovers a significant gap between current industry practices and regulatory expectations. A key bottleneck identified is insufficient collaboration between technical and legal teams. To bridge this gap, the study advocates for the development of integrated data quality tools and stronger cross-disciplinary collaboration to support the compliant deployment of trustworthy AI systems.

This work addresses the propagation of label errors in data validation, which can severely compromise the reliability of downstream query results. To quantify the impact of such errors and identify high-risk tuples whose uncertainty may be exacerbated by validation, the authors propose Maximum Error Score (MES)—a data-distribution-agnostic metric. Building on MES, they design MESReduce, an interactive validation optimization algorithm that adaptively guides the verification process by efficiently computing MES and incorporating feedback from external validators. Experimental evaluation on both real-world and synthetic datasets demonstrates that MESReduce significantly reduces the maximum error score and effectively enhances validation accuracy.