Early Stopping Against Label Noise Without Validation Data

2025-02-11

International Conference on Learning Representations

AI Summary

To address the performance degradation and model selection bias caused by validation-set-dependent early stopping under label noise, this paper proposes Label Wave—a validation-free, noise-robust early stopping method. Label Wave monitors the dynamic fluctuations of per-sample prediction confidence on the training set, revealing a strong correlation between minima in these fluctuations and the optimal generalization point. It automatically terminates training via sliding-window quantification and adaptive peak detection. As the first fully validation-free, noise-aware early stopping framework, it establishes a novel link between training dynamics (specifically, confidence fluctuation patterns) and generalization capability. Extensive experiments on benchmarks including CIFAR-10/100 and WebVision demonstrate that Label Wave consistently improves the accuracy of state-of-the-art noisy-label learning methods by 3.2–5.7% on average, while exhibiting superior stability compared to conventional validation-based early stopping.

Technology Category

• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Computer Vision: Adversarial Attacks & Robustness
• Search and Optimization: Learning to Search

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Early stopping methods in deep learning face the challenge of balancing the volume of training and validation data, especially in the presence of label noise. Concretely, sparing more data for validation from training data would limit the performance of the learned model, yet insufficient validation data could result in a sub-optimal selection of the desired model. In this paper, we propose a novel early stopping method called Label Wave, which does not require validation data for selecting the desired model in the presence of label noise. It works by tracking the changes in the model's predictions on the training set during the training process, aiming to halt training before the model unduly fits mislabeled data. This method is empirically supported by our observation that minimum fluctuations in predictions typically occur at the training epoch before the model excessively fits mislabeled data. Through extensive experiments, we show both the effectiveness of the Label Wave method across various settings and its capability to enhance the performance of existing methods for learning with noisy labels.

Problem

• Early stopping without validation data
• Handling label noise in training
• Preventing overfitting to mislabeled data

Innovation

• Label Wave early stopping
• No validation data required
• Tracks prediction changes