corpus curation

To address the unstructured nature of web-based pretraining data—which leads to content uncontrollability and labor-intensive curation—this paper proposes WebOrganizer, a novel framework introducing the first “topic + format” dual-dimensional domain taxonomy for structured organization and automatic annotation of web pages. Methodologically, it (i) builds an LLM-distillation-driven lightweight classifier; (ii) designs a domain-aware data mixing strategy; (iii) develops taxonomy-guided automatic annotation; and (iv) quantifies, for the first time, how quality filtering shifts implicit domain distributions, demonstrating the synergistic benefits of joint domain modeling and quality filtering. Experiments show that WebOrganizer significantly improves downstream task performance and enhances existing quality filtering efficacy, establishing a new, interpretable, and controllable paradigm for large language model pretraining data governance.

Existing corpus construction methods produce only flat document collections, lacking systematic knowledge organization and thus failing to meet the demand of large language models for high-quality structured data. This work proposes the CORTEX framework, which introduces the first three-layer heterogeneous Ontology-based Corpus Graph (OCG), comprising a quality-optimized content layer, a lightweight ontology layer evolved via LLM-driven mechanisms, and a cross-domain alignment layer. This architecture enables cross-domain associations at arbitrary granularities and supports automatic ontology evolution. Leveraging this framework, we construct a structured, high-quality corpus of 24.14 billion tokens and release CortexBench, a benchmark for cross-domain retrieval and reasoning, demonstrating its effectiveness across eight state-of-the-art large language models.

This work investigates the syntactic fidelity of large language models (LLMs) in text regeneration—specifically, their ability to preserve human-authored syntactic properties. Method: Focusing on Wikipedia and news corpora—the two canonical training domains—we systematically assess multi-level syntactic features—including sentence length, readability, dependency label distributions, parsing depth, and tree complexity—under semantic controllability constraints. We adopt a corpus-linguistic observational paradigm, employing open-source LLMs with prompt-driven regeneration, spaCy/CoreNLP parsing, and multidimensional syntactic statistical analysis. Contribution/Results: We provide the first quantitative evidence of systematic syntactic distributional shifts in LLM-regenerated text. Results reveal pervasive mean shifts, variance compression, and long-tail attenuation across nearly all syntactic dimensions—indicating significant distributional degradation. These findings demonstrate that LLMs fundamentally fail to replicate the syntactic diversity characteristic of human language.

This study addresses the cross-domain, cross-model detection of large language model (LLM)-generated text—specifically, robust identification under a setting where training and test domains fully overlap yet span multiple domains and diverse LLM sources. Building upon the RAID benchmark, we present the first systematic validation of high-accuracy detection feasibility under fixed, diversified domain–LLM combinations. We propose a unified framework integrating feature extraction, discriminative classification, and domain adaptation to jointly process multi-source generated texts. Our approach incorporates BERT/RoBERTa-based encoders, contrastive learning, and ensemble strategies. Among 23 submissions from 9 teams, several models achieved >99% accuracy with false positive rates ≤5%. These results surpass conventional single-domain or open-domain detection paradigms, demonstrating both the practical viability and scalability of cross-domain, cross-model detection for LLM-generated content.

This study addresses the lack of systematic approaches for constructing, storing, and sharing high-quality annotated corpora. It proposes a generalizable and reusable end-to-end methodology encompassing annotation guideline development, corpus annotation, data storage, sharing mechanisms, and value realization, with an emphasis on full lifecycle management and cross-domain applicability. Integrating linguistic annotation theory, data management standards, and collaborative research practices, the approach is articulated through a structured framework and illustrative examples to yield a clear and actionable guide. The resulting methodology provides standardized support for diverse research domains, significantly enhancing the efficiency and quality with which researchers can build and utilize annotated textual data.

This study addresses the linguistic barriers impeding the global dissemination of scientific research, which generic machine translation systems struggle to overcome due to their inability to accurately handle domain-specific terminology and complex syntactic structures in scholarly texts. To bridge this gap, the authors present the first systematic construction of Spanish–English, French–English, and Portuguese–English parallel and monolingual corpora spanning four scientific subfields: cancer, energy, neuroscience, and transportation. Leveraging these resources, they perform domain-adaptive fine-tuning of neural machine translation models. Experimental results demonstrate that the fine-tuned systems significantly outperform generic baselines in translation quality for scientific content, thereby confirming the critical role of multilingual, multidisciplinary specialized corpora in enhancing the accuracy and fluency of research literature translation.

This work addresses the challenge of efficiently and accurately detecting whether a given text is partially or fully contained—particularly via near verbatim copying—within massive web-scale corpora. To this end, the authors introduce FindMyText, an open-source tool that leverages document fingerprinting with a novel mechanism for identifying contiguous matching fingerprint sequences, explicitly capturing near-exact copied segments rather than relying on holistic text similarity. The system incorporates a distributed disk-based index to enable scalable processing. The study also establishes the first benchmark specifically designed for text containment tasks, demonstrating that FindMyText significantly outperforms existing methods across diverse datasets including arXiv, Wikipedia, and general web corpora, thereby validating its efficiency, robustness, and practical utility.

This study addresses the challenges of losing diverse perspectives and lacking traceability in the analysis of large-scale heterogeneous textual corpora. To this end, it proposes a structured reading approach grounded in large language models, which defers irreversible information compression by sequentially performing insight extraction, semantic clustering, theme generation, and iterative omission detection. This pipeline explicitly preserves divergent viewpoints, thereby enhancing both coverage and auditability of the analytical process. Evaluated on a corpus of 152 industrial policy documents, the method successfully extracted over 17,500 structured insights and constructed a comprehensive thematic map. The implementation has been open-sourced as the first end-to-end framework supporting large-scale qualitative synthesis.

This study addresses the challenge of legally sharing copyrighted annotated corpora, which hinders natural language processing models from capturing the full diversity of real-world data. To overcome this limitation, the authors propose a corpus distribution mechanism based on non-invertible hashing: corpus creators publicly release hashed versions of both source text and annotations, enabling users to recover the annotations by aligning their own licensed copies of the text through the same hash function. The approach integrates text alignment algorithms with version-tolerant strategies to achieve robust cross-version matching and is implemented in an open-source Python toolkit named novelshare. Experiments on multiple editions of novels demonstrate token-level alignment accuracy ranging from 98.7% to 99.79%, confirming the method’s efficiency and practical utility.

This study investigates whether large language models (LLMs) genuinely benefit from targeted cross-domain method transfer in research idea generation or merely improve due to exposure to diverse technical seeds. To address this, we propose PaperGym, a three-stage framework comprising tool-augmented seed extraction, paraphrase-based retrieval across seven machine learning domains, and rule-based synthetic evaluation. Through controlled experiments, we present the first systematic assessment of cross-domain retrieval’s effectiveness for LLM-driven ideation. Results show that cross-domain retrieval yields ideas significantly more novel than those from no-retrieval or within-domain baselines, yet performs comparably to random diverse seeds. This suggests that LLMs do not effectively leverage the semantic structure of retrieved content, highlighting fundamental limitations in current approaches to cross-domain knowledge transfer.