OpenLanguageModel: Readable and Composable Small-Language-Model Pretraining for Education and Research

2026-07-18

AI Summary

This work addresses the lack of readable, well-structured, and reusable open-source pretraining frameworks for small language models in education and research by introducing a modular PyTorch-based library. The framework employs composable primitives—such as Block, Residual, Repeat, and Parallel—to ensure alignment between model code and architectural diagrams, enabling seamless transition from pedagogical examples to full-scale pretraining. It integrates streaming data processing, mixed-precision training, callback mechanisms, and single-node multi-GPU support, allowing architecture or component substitution without code modification. Experiments demonstrate that a 348M-parameter model achieves 90.6% weak scaling efficiency across four GPUs, closely matching reference implementations. The project includes 27 preset models, comprehensive documentation, and has received positive early community feedback, effectively bridging teaching, research, and engineering practices.

Abstract

OpenLanguageModel (OLM) is an open-source PyTorch library for building and pretraining small language models while keeping their machinery visible. In OLM, model code reads like the architecture: components are ordinary modules, while Block, Residual, Repeat, and Parallel describe how they are wired. The resulting model can move unchanged from a teaching notebook to a complete pretraining run or a research ablation. OLM connects this readable model layer to tokenizers, local and streaming datasets, optimization, mixed precision, callbacks, checkpoints, and hardware-aware CPU, single-GPU, and single-node multi-GPU execution. We demonstrate the full path by tracing GPT-2 from diagram to code, launching a FineWeb-Edu training script, replacing one attention component, and letting AutoTrainer configure the available machine. The package includes 27 presets across nine familiar model families and documentation that progresses from LM fundamentals to architecture research. Validation shows close agreement with independent reference implementations, 90.6% four-GPU weak-scaling efficiency for a 348M-parameter workload, compact architecture edits, and positive early usability results. OLM is MIT-licensed and available through PyPI, GitHub, and its documentation site.

Problem

• small language models
• readable code
• composable architecture
• education
• research

Innovation

• readable modeling
• composable architecture
• small language models
• open-source pretraining
• hardware-aware training