SubRegWeigh: Effective and Efficient Annotation Weighing with Subword Regularization

2024-09-10

International Conference on Computational Linguistics

AI Summary

To address inevitable label noise in human-annotated NLP data, this paper proposes a single-pass, efficient, and robust annotation weighting method. The core innovation lies in the first introduction of subword regularization for label noise detection, synergistically integrating token-level perturbations with uncertainty modeling to enable fine-grained, token-level confidence estimation—without requiring ensemble models or iterative retraining. Empirically, the method significantly enhances model robustness against erroneous labels, delivering consistent performance gains on document classification and named entity recognition tasks; it further achieves high accuracy in identifying anomalous annotations under controlled pseudo-noise settings. Compared to conventional multi-model weighting approaches, it accelerates inference by 4–5×. The implementation is publicly available.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

NLP datasets may still contain annotation errors, even when they are manually annotated. Researchers have attempted to develop methods to automatically reduce the adverse effect of errors in datasets. However, existing methods are time-consuming because they require many trained models to detect errors. This paper proposes a time-saving method that utilizes a tokenization technique called subword regularization to simulate multiple error detection models for detecting errors. Our proposed method, SubRegWeigh, can perform annotation weighting four to five times faster than the existing method. Additionally, SubRegWeigh improved performance in document classification and named entity recognition tasks. In experiments with pseudo-incorrect labels, SubRegWeigh clearly identifies pseudo-incorrect labels as annotation errors. Our code is available at https://github.com/4ldk/SubRegWeigh .

Problem

• Natural Language Processing
• Data Error Correction
• Efficiency Improvement

Innovation

• SubRegWeigh
• subword regularization
• error mitigation