🤖 AI Summary
This work addresses the scarcity of large-scale, high-quality private code datasets in real-world industrial settings, which hinders the generalization of code intelligence research. To bridge this gap, we collaborate with twelve industry partners to construct the first industrial-scale private code dataset for research, comprising 2,440 repositories, 138 programming languages, and 373 million lines of code, accompanied by structured metadata and a deterministic anonymization pipeline. Crucially, the dataset aggregates authentic private code from critical domains such as fintech and enterprise applications through formal data-sharing agreements. Quality is ensured via a dual mechanism combining automated filtering and manual curation. This resource enables diverse applications including pretraining of code language models, software quality analysis, developer behavior studies, and the establishment of benchmarks for evaluating intelligent agents.
📝 Abstract
We present Curated Industrial Developer Repository (CIDR), a large-scale dataset of real-world software repositories collected through direct collaboration with 12 industrial partner organizations. The dataset comprises 2,440 repositories spanning 138 programming languages and totalling 373 million lines of code, accompanied by structured per-repository metadata. Unlike existing code corpora derived from public open-source platforms, CIDR consists exclusively of proprietary production codebases contributed under formal data sharing agreements, covering application domains including enterprise web and mobile development, fintech, and custom software consultancy. All repositories were processed through a multi-stage pipeline encompassing structured partner onboarding, two-stage quality selection combining automated metadata filtering with manual code review, and a deterministic anonymization pipeline covering the full version control history. The dataset is intended to support research in code intelligence, software quality analysis, pre-training and fine-tuning of code language models, developer behaviour studies, and construction of agent evaluation benchmarks. Access is provided under a restricted commercial license; details are available at https://fermatix.ai/#Contact.