🤖 AI Summary
In high-risk industrial applications, severe label noise critically degrades model performance and trustworthiness. To address this, we propose a knowledge graph (KG)-guided multi-LLM agent committee framework: it dynamically constructs sample-context KGs to orchestrate debate-style reasoning and consensus voting among multiple LLM agents for automated identification and precise correction of noisy labels. Our key innovation is a coverage-based neural-symbolic error correction logic, achieving 100% recall on structural errors and high-fidelity repair. On the AlleNoise subset, our method attains an F1 score of 0.99—substantially outperforming single-LLM baselines (0.48) and KG-free committee variants (0.59). This work establishes a novel, interpretable, and verifiable paradigm for data purification in high-reliability machine learning systems.
📝 Abstract
The performance of production machine learning systems is fundamentally limited by the quality of their training data. In high-stakes industrial applications, noisy labels can degrade performance and erode user trust. This paper presents Adjudicator, a system that addresses the critical data mining challenge of automatically identifying and correcting label noise and has been validated for production deployment. Adjudicator models this as a neuro-symbolic task, first constructing a dynamic Knowledge Graph (KG) to unify item context. This KG then informs a"Council of Agents,"a novel multi-agent Large Language Model architecture where specialized agents debate and vote on a label's validity. We validate our system on a 1,000-item balanced subset of the AlleNoise benchmark. Our KG-informed model achieves a 0.99 F1-score, significantly outperforming a single-LLM baseline (0.48 F1) and a non-KG council (0.59 F1). Our analysis reveals this is due to a Precision, achieved by a novel override logic that uses the KG to perfectly identify complex, structural errors (complete Recall) -- a class of errors that baselines fail to find. This result demonstrates a robust and explainable system for automated, high-precision data verification, serving as a vital proof-of-concept for generating golden datasets in strictly governed industrial environments.