A Pre-trained Data Deduplication Model based on Active Learning

2023-07-31

arXiv.org

AI Summary

Semantic-level duplicate detection in large-scale data remains challenging, and manual annotation incurs prohibitively high costs. Method: This paper proposes the first end-to-end deduplication model integrating active learning with pre-trained Transformers, reformulating deduplication as a sequence-to-classification task. It innovatively introduces active learning into semantic deduplication for the first time and designs an R-Drop–based enhancement strategy to improve the generalization capability of each annotation round. The approach unifies Transformer pre-training, active sampling, R-Drop regularization, and sequence classification fine-tuning. Results: On benchmark datasets, the method achieves a 28% improvement in Recall over existing state-of-the-art approaches, significantly reducing annotation effort while enhancing model robustness and generalization.

Abstract

In the era of big data, the issue of data quality has become increasingly prominent. One of the main challenges is the problem of duplicate data, which can arise from repeated entry or the merging of multiple data sources. These"dirty data"problems can significantly limit the effective application of big data. To address the issue of data deduplication, we propose a pre-trained deduplication model based on active learning, which is the first work that utilizes active learning to address the problem of deduplication at the semantic level. The model is built on a pre-trained Transformer and fine-tuned to solve the deduplication problem as a sequence to classification task, which firstly integrate the transformer with active learning into an end-to-end architecture to select the most valuable data for deduplication model training, and also firstly employ the R-Drop method to perform data augmentation on each round of labeled data, which can reduce the cost of manual labeling and improve the model's performance. Experimental results demonstrate that our proposed model outperforms previous state-of-the-art (SOTA) for deduplicated data identification, achieving up to a 28% improvement in Recall score on benchmark datasets.

Problem

• Big Data
• Duplicate Data
• Semantic Deduplication

Innovation

• Active Learning
• Transformer Model
• R-Drop Technology