Calibrating Pre-trained Language Classifiers on LLM-generated Noisy Labels via Iterative Refinement

2025-05-26

AI Summary

Large language models (LLMs) often generate noisy labels with low fidelity, degrading downstream classifier generalization. To address this, we propose SiDyP: a framework that retrieves plausible ground-truth label candidates via neighborhood label distribution analysis in the text embedding space, and calibrates noisy labels end-to-end—without human annotation—using a dynamic prior-guided simplex diffusion mechanism. This is the first systematic approach to mitigating LLM-generated label noise. SiDyP integrates neighborhood-aware label refinement, simplex-constrained diffusion modeling, and dynamic prior estimation, supporting both zero-shot and few-shot settings. Extensive evaluation across multiple LLMs (e.g., GPT-4, Claude) and diverse NLP tasks demonstrates that fine-tuning BERT classifiers with SiDyP-calibrated labels yields average accuracy improvements of 7.21% (zero-shot) and 7.30% (few-shot), significantly outperforming existing label denoising methods.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Diffusion Models for Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

The traditional process of creating labeled datasets is labor-intensive and expensive. Recent breakthroughs in open-source large language models (LLMs) have opened up a new avenue in generating labeled datasets automatically for various natural language processing (NLP) tasks, providing an alternative to such an expensive annotation process. However, the reliability of such auto-generated labels remains a significant concern due to inherent inaccuracies. When learning from noisy labels, the model's generalization is likely to be harmed as it is prone to overfit to those label noises. While previous studies in learning from noisy labels mainly focus on synthetic noise and real-world noise, LLM-generated label noise receives less attention. In this paper, we propose SiDyP: Simplex Label Diffusion with Dynamic Prior to calibrate the classifier's prediction, thus enhancing its robustness towards LLM-generated noisy labels. SiDyP retrieves potential true label candidates by neighborhood label distribution in text embedding space and iteratively refines noisy candidates using a simplex diffusion model. Our framework can increase the performance of the BERT classifier fine-tuned on both zero-shot and few-shot LLM-generated noisy label datasets by an average of 7.21% and 7.30% respectively. We demonstrate the effectiveness of SiDyP by conducting extensive benchmarking for different LLMs over a variety of NLP tasks. Our code is available on Github.

Problem

• Addressing reliability of LLM-generated noisy labels
• Improving model robustness against label inaccuracies
• Calibrating classifiers for better generalization performance

Innovation

• Uses simplex diffusion for label refinement
• Leverages neighborhood label distribution
• Improves BERT performance on noisy labels