software repository mining

Large language models (LLMs) face global copyright and contractual compliance risks stemming from pretraining data. To address this, we propose the first full-stack, open-source copyright-clean data ecosystem. Our system constructs a globally scalable, legally compliant training data pipeline comprising over 132 million human-verified documents, trillions of tokens, and 16 categories of trusted sources. Methodologically, we integrate automated license validation, multi-source heterogeneous document parsing, structured metadata modeling, and a CC-BY–compatible preprocessing architecture—enabling, for the first time, end-to-end legal verifiability across data acquisition, provenance tracking, and distribution. We release a complete open-source data stack—including raw documents, rich metadata, pre-tokenized sequences, and derivative resources (e.g., QA pairs, summaries, dialogues)—fully hosted on S3, Hugging Face, and GitHub for unrestricted global access.

This work addresses the scarcity of fine-grained, expert-annotated non-functional requirement (NFR) samples in existing public GitHub datasets by introducing GitReq—the first large-scale GitHub-based quality requirements dataset aligned with the ISO/IEC 25010 standard. GitReq comprises 6,302 expert-validated requirements extracted from 4,080 repositories, spanning all eight quality characteristics defined in the standard. The construction methodology employs category-specific tri-signal mining, a preprocessing step to separate functional from non-functional requirements, and a rigorous manual annotation protocol, achieving a Fleiss’ Kappa inter-annotator agreement of 0.72. Experimental evaluation demonstrates that GPT-5.2 attains a macro-averaged F1 score of 0.641 under zero-shot settings, confirming both the dataset’s validity and its inherent challenge for current language models.

This work addresses the limitations of existing software repository classification methods, which rely heavily on metadata such as README files and thus suffer from poor applicability in large-scale real-world scenarios due to frequent information gaps. To overcome this, we propose DRAGON, a novel approach that leverages only lightweight signals from version control systems—such as file and directory names—with optional integration of README content when available. DRAGON demonstrates remarkable robustness, with performance degrading by merely 6% in the absence of README files, and its predictions often yield semantically related “near-misses” that retain practical retrieval value. Evaluated on the largest open-source repository classification dataset to date, comprising 825,000 repositories, DRAGON achieves an F1@5 score of 60.8%, outperforming the current state-of-the-art by 6 percentage points and confirming its effectiveness and stability.

To address the low discoverability and usability of GitHub repositories caused by disorganized and incomplete README content, this paper proposes a fine-grained automatic classification method for README sections based on large language models (LLMs). Balancing performance and efficiency, we employ encoder architectures—including BERT, DistilBERT, and RoBERTa—integrated with the parameter-efficient fine-tuning technique LoRA, trained on a custom annotated dataset of 4,226 samples. Our framework achieves an overall F1-score of 0.98 on multi-class README section identification, significantly outperforming existing approaches. This work presents the first systematic validation of lightweight LLMs adapted via LoRA for structured parsing of open-source documentation. It demonstrates both technical feasibility and practical efficacy in enhancing repository searchability and supporting broader adoption of open-source projects.

This work addresses the scarcity of large-scale, high-quality private code datasets in real-world industrial settings, which hinders the generalization of code intelligence research. To bridge this gap, we collaborate with twelve industry partners to construct the first industrial-scale private code dataset for research, comprising 2,440 repositories, 138 programming languages, and 373 million lines of code, accompanied by structured metadata and a deterministic anonymization pipeline. Crucially, the dataset aggregates authentic private code from critical domains such as fintech and enterprise applications through formal data-sharing agreements. Quality is ensured via a dual mechanism combining automated filtering and manual curation. This resource enables diverse applications including pretraining of code language models, software quality analysis, developer behavior studies, and the establishment of benchmarks for evaluating intelligent agents.

Existing code documentation datasets suffer from high noise levels, pervasive redundancy, and severe contamination by AI-generated content, thereby degrading supervision signals and compromising evaluation validity. To address these issues, this work introduces the first quality-centric, function-level code documentation dataset, covering Python, Java, TypeScript, JavaScript, and C++. We propose a novel four-stage human-augmented cleaning pipeline integrating structural complexity filtering, near-duplicate detection, AI-generated content identification, type annotation analysis, and heuristic discrimination. The pipeline reduces the raw corpus to 25.6% high signal-to-noise ratio samples, achieving an average documentation quality score of 6.93/10, with 86.9% containing explicit type annotations and only 2.9% exhibiting AI-generation hallmarks. Fine-tuning LLMs on this dataset yields substantial improvements: BLEU increases by 29.47% and ROUGE-L by 24.04%, demonstrating its efficacy for robust code documentation modeling.

This work addresses the challenge of efficiently and accurately detecting whether a given text is partially or fully contained—particularly via near verbatim copying—within massive web-scale corpora. To this end, the authors introduce FindMyText, an open-source tool that leverages document fingerprinting with a novel mechanism for identifying contiguous matching fingerprint sequences, explicitly capturing near-exact copied segments rather than relying on holistic text similarity. The system incorporates a distributed disk-based index to enable scalable processing. The study also establishes the first benchmark specifically designed for text containment tasks, demonstrating that FindMyText significantly outperforms existing methods across diverse datasets including arXiv, Wikipedia, and general web corpora, thereby validating its efficiency, robustness, and practical utility.

This work addresses a critical gap in academic information extraction by shifting focus from scholarly papers to implementation-level research artifacts in code repositories. We formally define and annotate ten categories of implementation-level entities within README files, introducing NERdME—a high-quality named entity recognition dataset comprising 200 expert-annotated READMEs and over 10,000 entity spans. Leveraging this resource, we conduct baseline experiments with large language models and fine-tuned Transformers, revealing substantial differences between implementation-level and paper-level entities. Furthermore, we demonstrate the practical utility of our approach through downstream entity linking tasks, showing its effectiveness in research artifact discovery and metadata integration. This study thus establishes the first systematic framework for semantic information extraction from code repositories, filling a longstanding void in academic knowledge extraction.

Current data processing pipelines for post-training large language models—encompassing cleaning, deduplication, synthesis, and quality filtering—are fragmented and lack auditability and sample-level decision transparency. This work proposes the first end-to-end configurable data processing framework that unifies data ingestion, cleaning, LLM-driven synthesis across eight task types, three-tiered quality gating, and export modules. The system introduces sample-level provenance tracking and a precise hallucination verification mechanism. It supports six input formats and over 100 model APIs via LiteLLM, offering both a YAML-driven command-line interface and a Python API. Outputs are compatible with five training formats used by TRL, Unsloth, and AlignTune, substantially enhancing transparency, reproducibility, and scalability in post-training data preparation.

Current large code models exhibit limited performance in repository-level code generation due to their neglect of cross-file dependencies and structural context, while conventional NLP-based retrieval-augmented approaches struggle to effectively model the inherent structure of code. To address this, this work proposes Hydra, a novel framework that treats code as structured entities and introduces a hierarchical code tree index, a dependency-aware retriever (DAR), and a hybrid retrieval mechanism that integrates functional dependencies with semantic similarity. Hydra departs from traditional NLP-style paradigms for code processing and achieves state-of-the-art results on the DevEval and RepoExec benchmarks, surpassing the strongest baseline by over 5% in Pass@1. Notably, it enables smaller models equipped with Hydra to match the performance of larger models using conventional retrievers.