🤖 AI Summary
This study addresses the fragmentation of academic data across isolated databases, which suffers from inconsistent metadata and missing cross-source linkages. To overcome these challenges, the authors construct a locally deployable academic data lake based on DuckDB and Parquet, integrating eight major open scholarly sources and harmonizing approximately 293 million publications through DOI standardization. They introduce a novel cross-source unified query architecture and propose an innovative scientific ontology alignment method leveraging BGE-large embeddings, achieving an F1 score of 0.77 while covering 99.8% of research topics—significantly outperforming traditional text-matching baselines. The resulting dataset spans 960 GB, encompasses 22 schemas and 153 SQL views, and undergoes rigorous multi-dimensional validation (both automated and manual), enabling sophisticated cross-source scholarly analysis.
📝 Abstract
Scholarly data are largely fragmented across siloed databases with divergent metadata and missing linkages among them. We present the Science Data Lake, a locally-deployable infrastructure built on DuckDB and simple Parquet files that unifies eight open sources - Semantic Scholar, OpenAlex, SciSciNet, Papers with Code, Retraction Watch, Reliance on Science, a preprint-to-published mapping, and Crossref - via DOI normalization while preserving source-level schemas. The resource comprises approximately 960GB of Parquet files spanning ~293 million uniquely identifiable papers across ~22 schemas and ~153 SQL views. An embedding-based ontology alignment using BGE-large sentence embeddings maps 4,516 OpenAlex topics to 13 scientific ontologies (~1.3 million terms), yielding 16,150 mappings covering 99.8% of topics ($\geq 0.65$ threshold) with $F1 = 0.77$ at the recommended $\geq 0.85$ operating point, outperforming TF-IDF, BM25, and Jaro-Winkler baselines on a 300-pair gold-standard evaluation. We validate through 10 automated checks, cross-source citation agreement analysis (pairwise Pearson $r = 0.76$ - $0.87$), and stratified manual annotation. Four vignettes demonstrate cross-source analyses infeasible with any single database. The resource is open source, deployable on a single drive or queryable remotely via HuggingFace, and includes structured documentation suitable for large language model (LLM) based research agents.