language identification

To address the scarcity of high-quality, large-scale, fine-grained annotated data for multilingual large language models and machine translation research, this work introduces the first open-source, large-scale multilingual text data construction framework. It supports nearly 200 languages and scales to 30 trillion tokens, integrating end-to-end techniques including web page cleaning, noise-robust language identification, exact and fuzzy deduplication, PII detection, registry label annotation, text quality scoring, and synthetic parallel corpus generation. We propose the first native-task-based multilingual evaluation framework, releasing standardized benchmarks across nine languages and an automated assessment pipeline. The resulting dataset constitutes the largest publicly available multilingual pretraining corpus to date. Using it, we train 57 monolingual encoder-decoder models and multiple GPT-style monolingual models. All tooling—including data processing pipelines, evaluation benchmarks, and pretrained model families—is fully open-sourced.

This study addresses the critical challenges in Lombard, a low-resource language, where existing NLP corpora suffer from widespread mislabeling, templated content, noise, and severe dialectal imbalance. For the first time, the authors conduct a systematic manual audit, integrating language identification validation, orthographic analysis, and dialect classification to assess linguistic authenticity and regional representativeness. Their findings reveal that mainstream datasets contain an extremely low proportion of genuine Lombard texts, exhibit inconsistent orthography, and display a pronounced representational bias favoring Western dialects while marginalizing Eastern varieties. The work underscores the necessity of moving beyond quantity-driven data collection toward community-informed, dialect-sensitive strategies for building high-quality linguistic resources, offering crucial methodological guidance for corpus development in other low-resource languages.

This study addresses the performance degradation commonly observed in multilingual large language models, which stems from imbalanced data distributions and the so-called “curse of multilinguality.” The authors identify the root cause as remediable corpus quality issues and propose a language-specific data curation and balancing strategy. By integrating multilingual quality evaluation with an efficient training mixture methodology, they optimize the composition of a 20-trillion-token corpus. Models trained on this refined dataset—specifically 3B and 8B parameter variants—achieve state-of-the-art multilingual performance while using 4–10 times fewer FLOPs than competing approaches. Furthermore, the curated corpus significantly enhances the multilingual scaling efficiency of Trinity Large (400B), demonstrating its effectiveness in improving both model performance and training efficiency across diverse languages.

To address the challenges of poor generalizability in preprocessing pipelines and uneven data quality in multilingual large language model (LLM) training, this work introduces the first scalable, automated multilingual pretraining data processing framework—supporting up to one thousand languages. Built upon Common Crawl, the framework integrates language-aware efficient filtering, cross-lingual deduplication, and joint optimization of duplication rate and quality for data rebalancing. It employs end-to-end evaluation via ablation studies guided by multilingual downstream tasks. We release FineWeb2, a 5-billion-document, 20-TB multilingual dataset covering nine languages. Experiments demonstrate substantial improvements in non-English LLM performance across multiple benchmarks. This work establishes a systematic, reproducible infrastructure for high-quality multilingual foundation model training.

Low-resource languages suffer from a critical scarcity of high-quality multilingual textual data, severely constraining the development of large language models. To address this, we propose the first systematic, open-source web corpus construction framework specifically designed for low-resource languages. Our framework integrates multi-stage collaborative processing: adaptive web page extraction, language-aware cleaning, cross-document semantic deduplication, fine-grained safety filtering, multidimensional quality assessment, and topic-consistency classification—ensuring both linguistic diversity and enhanced data security and reliability. We publicly release high-quality corpora covering five low-resource languages. Empirical evaluations demonstrate superior data quality, safety, and usability compared to existing benchmarks. The datasets and code are fully open-sourced on OpenDataLab and GitHub, providing a robust, scalable foundation for multilingual large model training and research.

Arabic pretraining corpora suffer from severe redundancy (nearly 60% token-level duplication) and heterogeneous quality. To address this, we propose a “data reuse over new crawling” paradigm, systematically integrating seven existing public Arabic web datasets. Our pipeline applies Arabic-specific quality filtering, MinHash-based deduplication at both document and sentence levels, multi-source fusion, and metadata alignment. The resulting corpus—currently the largest publicly available, deeply deduplicated Arabic dataset—comprises 178 billion tokens across 179 million documents. Empirical evaluation demonstrates substantial improvements in downstream model training efficiency and generalization performance. This corpus has become the de facto standard training data for multiple open-source Arabic large language models, establishing a new principle in Arabic NLP: rigorous, quality-driven data curation takes precedence over mere scale expansion.

Pretraining data for Romanian is scarce, heterogeneous in quality, and insufficiently diverse in topical coverage. Method: This paper proposes a lightweight multi-task filtering framework that jointly evaluates educational value, performs topic modeling, and analyzes syntactic and formatting diversity to conduct hierarchical quality screening of LLM-generated annotated texts. Unlike conventional unidimensional data cleaning, the framework systematically characterizes cross-lingual disparities between Romanian and English pretraining corpora across topic distribution, pedagogical relevance, and structural diversity. Contribution/Results: Empirical evaluation demonstrates that models trained on the filtered Romanian corpus achieve substantial performance gains on downstream tasks—including ROBUST and RONEC—validating the efficacy of structured, domain-aware data curation for low-resource language modeling. The work establishes a principled paradigm for small-language pretraining data construction, highlighting its critical role in enhancing model capabilities.

This study addresses the scarcity of multilingual corpora supporting emerging concepts—such as “non-technological innovation”—in the humanities and social sciences (HSS). To tackle this, we propose a hybrid multilingual corpus construction methodology that integrates corporate websites and annual reports, combining automatic language identification, domain-adapted content filtering, relevant paragraph extraction, expert-lexicon-driven contextual block identification, thematic annotation, and enriched structured metadata. Our key contribution is the first systematic construction of a high-quality, multilingual corpus specifically designed for HSS emerging concepts, accompanied by a parallel English supervised dataset with fine-grained thematic labels. The resulting resource enables cross-lingual lexical variation analysis, training of multilingual NLP models, and empirical social science research. It is both reusable and extensible, effectively bridging a critical gap between domain-specific knowledge modeling and computational linguistics applications.

This work addresses the challenge of low-resource machine translation, where performance is hindered by the scarcity of high-quality parallel corpora. To this end, the authors propose LALITA, a novel framework that systematically leverages lexical and linguistic features of source sentences—such as syntactic complexity—to identify and select high-value training samples. Combined with synthetic data augmentation, this approach significantly reduces data requirements while improving translation quality. The method is evaluated across multiple low-resource languages, including Hindi, Odia, Nepali, Nynorsk, and German, consistently yielding performance gains across training sets ranging from 50K to 800K sentence pairs. Notably, LALITA achieves these improvements with over 50% less training data, demonstrating both its efficiency and strong generalization capability.

This work addresses the challenge of efficiently and accurately detecting whether a given text is partially or fully contained—particularly via near verbatim copying—within massive web-scale corpora. To this end, the authors introduce FindMyText, an open-source tool that leverages document fingerprinting with a novel mechanism for identifying contiguous matching fingerprint sequences, explicitly capturing near-exact copied segments rather than relying on holistic text similarity. The system incorporates a distributed disk-based index to enable scalable processing. The study also establishes the first benchmark specifically designed for text containment tasks, demonstrating that FindMyText significantly outperforms existing methods across diverse datasets including arXiv, Wikipedia, and general web corpora, thereby validating its efficiency, robustness, and practical utility.