error annotation

This work addresses the challenge of effectively reusing failure feedback from existing agent execution trajectories, which are often lengthy, instance-specific, and lack standardized failure descriptions. To overcome this, the authors propose an unsupervised method that automatically distills raw trajectories into a structured, evidence-backed failure taxonomy. This taxonomy forms an adaptive failure glossary organized along three axes—system-level, role-level, and domain-level—and serves as a unified feedback interface integrated into trajectory selection, runtime monitoring, and system search processes. Requiring no manual annotation, the glossary achieves a 10× compression ratio while exhibiting semantics closely aligned with expert annotations. Empirical results demonstrate significant performance gains across multiple benchmarks: SWE-agent’s resolution rate improves from 60% to 70%, Claude Code reaches 70.7%, and Terminal-Bench 2.0 accuracy increases by 8–15 points.

Widespread label noise (10–25%) in NLP benchmark datasets leads to systematic underestimation of model performance, with many purported “LLM failures” attributable to annotation errors rather than model limitations. Method: We propose LLM-as-a-judge—a framework leveraging ensemble judgments from GPT-4, Claude, and Llama, combined with consistency voting and error-sensitivity analysis to automatically detect mislabeled instances; we further apply label smoothing and confident learning for robust label recalibration. Contribution/Results: Comprehensive evaluation across the TRUE benchmark suite reveals substantial disparities in quality and efficiency among expert, crowdsourced, and LLM-generated annotations. After correction, state-of-the-art models achieve average accuracy gains of 3.2–7.8 percentage points. This work provides the first empirical evidence of systematic label-noise interference in LLM evaluation and introduces a scalable, collaborative adjudication paradigm that reframes data correction as model performance recalibration.

This work addresses the instability of large language models (LLMs) in subjective annotation tasks—such as those involving psychological constructs—and the inadequacy of conventional alignment metrics in distinguishing error types and their downstream impacts. The authors propose a diagnostic evaluation framework that decouples LLM annotation errors along two dimensions: source (inherent task ambiguity versus model-induced error) and error type, introducing a novel error taxonomy tailored for ordinal subjective tasks. By integrating lightweight human audits, computational decomposition, and human–AI collaborative analysis, the approach is validated across four educational annotation tasks. It not only enables low-cost assessment of task suitability for LLM annotation but also offers actionable pathways for model refinement.

In autonomous driving AI perception system (AIePS) development, annotation quality critically impacts model safety and reliability; however, empirical understanding of how annotation errors originate and propagate across multi-organizational automotive supply chains remains lacking. This study addresses this gap by adopting dual perspectives—annotation lifecycle and supply chain—through semi-structured interviews with 20 domain experts from six organizations (50 hours total) and six-phase thematic coding analysis. We propose the first annotation defect taxonomy for AI perception systems, comprising 18 error types organized along three dimensions: completeness, accuracy, and consistency. Analogous to Failure Mode and Effects Analysis (FMEA), this taxonomy serves as a “failure mode library” for annotation defects. Validated by industry practitioners, it supports root-cause analysis, supplier evaluation, annotator onboarding, and annotation guideline refinement—thereby enhancing AIePS development reliability and safety.

The underlying mechanisms of code generation errors in large language models (LLMs) remain poorly understood. Method: Leveraging the HumanEval benchmark, this work systematically analyzes errors produced by six state-of-the-art LLMs and introduces, for the first time, a multidimensional, fine-grained error taxonomy integrating both semantic and syntactic dimensions. Using open coding and thematic analysis—augmented by statistical testing and qualitative root-cause attribution—the study identifies over ten recurrent error patterns, including logical flaws, boundary condition failures, and API misuse. Contribution/Results: The analysis reveals that LLM errors exhibit nontriviality, cross-line dependencies, and dispersed distribution—uncovering latent, deep-seated errors even in high-pass-rate tasks. It further demonstrates a nonlinear positive correlation between error frequency and task complexity. This taxonomy provides an interpretable, extensible theoretical foundation and empirical grounding for error localization, diagnosis, and repair in LLM-generated code.

This study addresses the widespread neglect in machine learning research of when validation occurs during data annotation—a critical factor influencing both label quality and cost—despite overreliance on post-hoc quality control. Drawing inspiration from the “shift-left” principle in software engineering, this work proposes a tripartite classification of quality checkpoints across early, intermediate, and late stages of the annotation pipeline and introduces a parameterized error propagation model that, for the first time, treats validation timing as a quantifiable design variable. Through error propagation modeling, process decomposition, and literature analysis, the authors find that only 4% of recent studies report validation timing. Their analysis demonstrates that early-stage quality checks can reduce error correction costs by up to two orders of magnitude. The paper calls for standardized reporting of timing configurations, platform support for tunable timing parameters, and empirical studies on stage-specific detection rates.

This study addresses the frequent failure of large language models (LLMs) in software engineering tasks due to structurally invalid outputs—such as syntactic or formatting errors—that prevent correct parsing by downstream toolchains, even when the semantic content is accurate. The authors systematically evaluate the reliability of structured output generation across four representative tasks, categorizing errors into syntactic, structural, and semantic types. They propose a template-driven token-matching generation (TTMG) method that enforces structural consistency during autoregressive decoding. Experimental results demonstrate that TTMG nearly eliminates syntactic errors; however, structural and semantic errors remain prevalent. This work reveals, for the first time, that the fundamental bottleneck in structured output generation lies not merely in syntax but in the insufficient coordination between structural and semantic correctness, indicating that existing structural control mechanisms, while necessary, are insufficient without joint guarantees of both dimensions.

Current automated formalization evaluations lack interpretable diagnostics for semantic errors, hindering both system optimization and human understanding. This work proposes FormalRx, a novel framework that introduces the first fine-grained, hierarchical taxonomy of Semantic Correctness Issues (SCI) comprising 28 error categories, and develops an end-to-end diagnostic model, FormalRx-8B, capable of aligning, classifying, localizing, and correcting formalization errors. Trained on 56,287 fine-grained annotated samples, the model achieves strong performance across four diagnostic tasks—0.88 F1 for alignment, 0.71 F1 for classification, 0.75 accuracy for localization, and 0.73 accuracy for correction—significantly outperforming both general-purpose large language models and specialized baselines. The study also releases FormalRx-Test, the first fine-grained diagnostic benchmark, thereby establishing a closed-loop pipeline from opaque evaluation to actionable feedback.

This work addresses the challenge posed by frequent and large-scale code changes in modern software projects, which overwhelm traditional code review practices. While existing large language model (LLM)-based approaches primarily focus on generating summaries, they lack structured identification of change types. To bridge this gap, the paper proposes a two-stage pipeline that leverages LLMs to perform taxonomy-based structured labeling of code diffs and extract semantic relationships and attributes—such as rename propagation and type modifications. This approach represents the first systematic exploration of LLMs for structured understanding of code changes, operating without reliance on static analysis toolchains and supporting language-agnostic, customizable taxonomies. Evaluated on both natural and synthetic patch benchmarks, the best configuration achieves 84% recall and 81% precision, with notably high accuracy in metadata extraction.

This work addresses the high cost and substantial cognitive load associated with structured annotation, which hinder efficient allocation of labeling efforts among heterogeneous annotators such as humans and models. The authors propose a center-theory-based task decomposition approach that identifies semantic centers to constrain the output space, formally models reasoning load, and introduces an algorithm for allocating heterogeneous annotation resources. Notably, this is the first method to integrate task decomposition with annotator capability matching. Experimental results demonstrate that the proposed approach significantly reduces cognitive burden while simultaneously improving both annotation quality and cost efficiency under a fixed budget.