🤖 AI Summary
This work addresses the scarcity of legally compliant, reproducible, and high-quality full-text corpora in chemistry suitable for retrieval and text mining. The authors propose a reproducible construction pipeline that integrates legal compliance screening with technical validation, building upon the S2ORC dataset. By leveraging Unpaywall, OpenAlex, and Crossref to identify papers under open licenses, they assemble a corpus of over 580,000 chemistry articles. For each document, they generate paragraph-level embeddings using e5-large-v2, produce machine-generated summaries, and assign multi-label subfield annotations. The project releases complete code, metadata, and provenance artifacts to enable transparent reconstruction. Technical validation confirms high standards in text quality, embedding consistency, and metadata completeness, substantially enhancing the accessibility and reusability of chemical literature resources.
📝 Abstract
We present Lit2Vec, a reproducible workflow for constructing and validating a chemistry corpus from the Semantic Scholar Open Research Corpus using conservative, metadata-based license screening. Using this workflow, we assembled an internal study corpus of 582,683 chemistry-specific full-text research articles with structured full text, token-aware paragraph chunks, paragraph-level embeddings generated with the intfloat/e5-large-v2 model, and record-level metadata including abstracts and licensing information. To support downstream retrieval and text-mining use cases, an eligible subset of the corpus was additionally enriched with machine-generated brief summaries and multi-label subfield annotations spanning 18 chemistry domains. Licensing was screened using metadata from Unpaywall, OpenAlex, and Crossref, and the resulting corpus was technically validated for schema compliance, embedding reproducibility, text quality, and metadata completeness. The primary contribution of this work is a reproducible workflow for corpus construction and validation, together with its associated schema and reproducibility resources. The released materials include the code, reconstruction workflow, schema, metadata/provenance artifacts, and validation outputs needed to reproduce the corpus from pinned public upstream resources. Public redistribution of source-derived text and broad text-derived representations is outside the scope of the general release. Researchers can reproduce the workflow by using the released pipeline with publicly available upstream datasets and metadata services.