On Revisiting Entropy for Identifying Mislabeled Images

2026-05-29

AI Summary

This work addresses the challenge of label noise in training data, which over-parameterized deep models tend to memorize, thereby degrading performance. The authors propose a novel statistic, Signed Entropy Integral (SEI), that effectively identifies mislabeled samples by jointly capturing both the magnitude of prediction entropy and its dynamic trend during training: correctly labeled samples exhibit consistently decreasing entropy, whereas mislabeled ones maintain high entropy throughout. SEI requires no additional training overhead and is readily applicable to diverse classification architectures, demonstrating particular efficiency within CLIP-based models. Evaluated on four medical imaging datasets, SEI significantly outperforms existing methods, achieving state-of-the-art performance in label error detection while maintaining remarkable simplicity and practicality.

Abstract

Mislabeled samples in training datasets severely degrade the performance of deep networks, as overparameterized models tend to memorize erroneous labels. We address this challenge by proposing a novel approach for mislabeled data detection that leverages training dynamics. Our method is grounded in the key observation that correctly labeled samples exhibit consistent entropy decrease during training, while mislabeled samples maintain relatively high entropy throughout the training process. Building on this insight, we introduce a signed entropy integral (SEI) statistic that captures both the magnitude and temporal trend of prediction entropy across training epochs. SEI is broadly applicable to classification networks and demonstrates particular effectiveness when integrated with contrastive language-image pretraining (CLIP) architectures. Through extensive experiments on four medical imaging datasets -- a domain particularly susceptible to labeling errors due to diagnostic complexity -- spanning diverse modalities and pathologies, we demonstrate that SEI achieves state-of-the-art performance in mislabeled data identification, outperforming existing methods while maintaining computational efficiency and implementation simplicity. Our code is available at https://github.com/MedAITech/SEI.

Problem

• mislabeled images
• entropy
• training dynamics
• data quality
• deep learning

Innovation

• signed entropy integral
• mislabeled data detection
• training dynamics
• prediction entropy
• CLIP