Domain-Adaptive Continued Pre-Training of Small Language Models

2025-04-13

AI Summary

Adapting small language models (125M parameters) to the education domain under constrained computational resources remains challenging due to high training costs and catastrophic forgetting during fine-tuning. Method: This paper proposes a continual pre-training paradigm tailored for small models, leveraging 0.4–1.0 billion tokens of domain-specific educational text. It integrates data cleaning, gradient checkpointing, and mixed-precision training to optimize memory usage while preserving general linguistic capabilities and enhancing domain specialization. Contribution/Results: The approach achieves superior token efficiency and domain adaptation without full retraining. On multi-benchmark evaluations—MMLU and HellaSwag—it improves accuracy by 8.1% and 7.6%, respectively, outperforming standard baselines. Empirical results demonstrate that lightweight continual pre-training is a viable, computationally efficient alternative for domain adaptation of small models in resource-constrained settings, offering a practical pathway for building domain-specific language models in education.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Natural Language Processing: (Large) Language Models
• Application Domains: Humanities & Computational Social Science

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Vertical and domain-specific search

Abstract

Continued pre-training of small language models offers a promising path for domain adaptation with limited computational resources. I've investigated this approach within educational domains, evaluating it as a resource-efficient alternative to training models from scratch. Using a 125M parameter model, I demonstrate significant performance improvements through incremental training on 400 million tokens, followed by further training to reach 1 billion tokens. My approach includes comprehensive data preprocessing, memory-optimized training configurations, and benchmark-based evaluation. Results show notable gains in knowledge-intensive tasks (MMLU +8.1%) and contextual understanding (HellaSwag +7.6%), while revealing educational domain specialization trade-offs. I analyze token efficiency, catastrophic forgetting mitigation strategies, and scaling patterns. My findings suggest that thoughtful preprocessing and training methodologies enable meaningful improvements in language model capabilities even with constrained computational resources, opening pathways for domain-specific adaptation of smaller language models.

Problem

• Domain adaptation of small language models with limited resources
• Improving performance via incremental training and data preprocessing
• Balancing educational specialization and general language capabilities

Innovation

• Domain-adaptive continued pre-training for small models
• Memory-optimized training with incremental token scaling
• Comprehensive preprocessing and benchmark-based evaluation