Infini-gram mini: Exact n-gram Search at the Internet Scale with FM-Index

2025-06-13

AI Summary

Exact n-gram matching and document provenance on internet-scale text (46 TB) face prohibitive storage overhead and computational bottlenecks. Method: We propose the first scalable, lightweight FM-index optimization system, integrating compression and indexing, enabling multithreaded parallel index construction, memory-aware query optimization, and a Web API service architecture. Contribution/Results: Our system reduces index size to 44% of the original text, accelerates indexing by 18×, and cuts memory consumption by 3.2×—enabling full-index construction on a single node in 50 days. We further introduce the first large-scale empirical framework for analyzing data contamination in large language models, revealing up to 40% training-set contamination in widely used benchmarks such as SQuAD. The system—including its API and interactive interface—is open-sourced to support transparent, reproducible contamination monitoring benchmarks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Distributed Search

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Language models are trained mainly on massive text data from the Internet, and it becomes increasingly important to understand this data source. Exact-match search engines enable searching in large text corpora -- counting string appearances and retrieving the enclosing documents -- yet the high storage overhead hinders their application on Internet-scale data. We present Infini-gram mini, an efficient and scalable system that can make petabyte-level text corpora searchable. Based on the FM-index data structure (Ferragina and Manzini, 2000), which simultaneously indexes and compresses text, our system creates indexes with size only 44% of the corpus. Infini-gram mini greatly improves upon the best existing implementation of FM-index in terms of indexing speed (18$ imes$) and memory use during both indexing (3.2$ imes$ reduction) and querying (down to a negligible amount). We index 46TB of Internet text in 50 days with a single 128-core CPU node (or 19 hours if using 75 such nodes). We show one important use case of Infini-gram mini in a large-scale analysis of benchmark contamination. We find several core LM evaluation benchmarks to be heavily contaminated in Internet crawls (up to 40% in SQuAD), which could lead to overestimating the capabilities of language models if trained on such data. We host a benchmark contamination bulletin to share the contamination rate of many core and community-contributed benchmarks. We also release a web interface and an API endpoint to serve general search queries on Infini-gram mini indexes.

Problem

• Enable exact n-gram search on Internet-scale text data
• Reduce storage overhead for large text corpora indexing
• Detect benchmark contamination in language model training data

Innovation

• FM-index for efficient Internet-scale text search
• 44% corpus size indexes with fast processing
• Scalable petabyte-level text search solution