Institutional Books 1.0: A 242B token dataset from Harvard Library's collections, refined for accuracy and usability

2025-06-10

AI Summary

High-quality, traceable training data for classical texts remains scarce, and sustainable governance mechanisms for such data are lacking. Method: We construct a 242-billion-token public-domain classical text dataset derived from the Harvard Library–Google Books partnership, covering 250+ languages and 983,000 volumes. We propose a sustainable data governance framework tailored to academic institutions’ digital heritage, enabling systematic quality assessment, automated copyright status classification, and end-to-end provenance chain construction across 1.07 million scanned volumes—the first such comprehensive effort. Data undergo OCR enhancement, multilingual standardization, hybrid rule- and model-based copyright identification, and fine-grained metadata modeling (including source, cataloging, and generative metadata). Contribution/Results: The fully open-sourced dataset significantly improves the usability, trustworthiness, and reproducibility of historical texts in LLM training and human–AI collaborative humanities research.

Technology Category

• Application Domains: Humanities & Computational Social Science
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity
• Security and Privacy: Data transparency and provenance

Abstract

Large language models (LLMs) use data to learn about the world in order to produce meaningful correlations and predictions. As such, the nature, scale, quality, and diversity of the datasets used to train these models, or to support their work at inference time, have a direct impact on their quality. The rapid development and adoption of LLMs of varying quality has brought into focus the scarcity of publicly available, high-quality training data and revealed an urgent need to ground the stewardship of these datasets in sustainable practices with clear provenance chains. To that end, this technical report introduces Institutional Books 1.0, a large collection of public domain books originally digitized through Harvard Library's participation in the Google Books project, beginning in 2006. Working with Harvard Library, we extracted, analyzed, and processed these volumes into an extensively-documented dataset of historic texts. This analysis covers the entirety of Harvard Library's collection scanned as part of that project, originally spanning 1,075,899 volumes written in over 250 different languages for a total of approximately 250 billion tokens. As part of this initial release, the OCR-extracted text (original and post-processed) as well as the metadata (bibliographic, source, and generated) of the 983,004 volumes, or 242B tokens, identified as being in the public domain have been made available. This report describes this project's goals and methods as well as the results of the analyses we performed, all in service of making this historical collection more accessible and easier for humans and machines alike to filter, read and use.

Problem

• Addresses scarcity of high-quality public LLM training data
• Introduces a curated 242B token dataset from Harvard Library
• Enhances accessibility of historical texts for AI and human use

Innovation

• Extracted and processed Harvard Library's public domain books
• Created a 242B token dataset with detailed documentation
• Improved accessibility and usability of historical texts