FOLD: Fuzzy Online Deduplication for Very Large Evolving Datasets via Approximate Nearest Neighbor Search

2026-06-01

AI Summary

This work addresses the poor scalability of existing fuzzy deduplication methods on large-scale dynamic data and their inability to support efficient online processing. To overcome these limitations, the authors propose FOLD, the first system to integrate HNSW indexing into online fuzzy deduplication, leveraging an incremental update mechanism that avoids full index reconstruction or exhaustive scanning. To enhance graph retrieval reliability, FOLD employs bitmap embeddings to mitigate score concentration under Jaccard similarity and combines this with approximate nearest neighbor search to achieve both high throughput and high recall. Experimental results demonstrate that FOLD attains recall rates of 93–97% on benchmark datasets including LM1B, C4, RealNews, and Common Crawl, while achieving up to a 2.09× improvement in throughput, significantly outperforming current state-of-the-art approaches.

Abstract

Fuzzy deduplication is key to constructing large language model training corpora. However, classic Locality-Sensitive Hashing pipelines scale poorly as corpora grow and are ill-suited to continuous ingestion. We present FOLD (Fuzzy Online Deduplication), an online fuzzy deduplication system that delivers high recall and throughput for evolving datasets. FOLD maintains an incrementally updated HNSW index over admitted documents, retrieving a small, high-quality candidate neighborhood for each incoming document instead of repeatedly rebuilding global buckets or rescanning the accumulated corpus. To our knowledge, FOLD is the first online fuzzy deduplication system to use HNSW. However, applying Jaccard similarity out of the box causes score crowding, making graph traversal unreliable within a small number of steps. FOLD addresses this with a bitmap representation that provides a more discriminative, Jaccard-aligned signal during HNSW search. Across four LLM-scale datasets (LM1B, C4, RealNews, and Common Crawl), FOLD stays fast and accurate as the corpus grows: at the largest evaluated scales, it maintains 93-97% recall and achieves up to 2.09x higher throughput than competing alternatives, whose best recall reaches only 76%.

Problem

• fuzzy deduplication
• large language models
• evolving datasets
• approximate nearest neighbor search
• Jaccard similarity

Innovation

• Fuzzy Deduplication
• Online Deduplication
• HNSW
• Approximate Nearest Neighbor Search
• Jaccard Similarity