🤖 AI Summary
To address the high computational cost, privacy risks, and sustainability challenges arising from the parameter- and data-scaling paradigm of large language models (LLMs), this work proposes a “small-and-specialized” paradigm. We introduce ELM—a lightweight, context-specific language model (900M parameters) tailored to higher education settings. Built upon the Transformer architecture, ELM is pre-trained and fine-tuned exclusively on institutional corpora from Erasmus University Rotterdam, incorporating a domain-specific vocabulary and task-adaptive output heads. Experimental results demonstrate that ELM matches the performance of general-purpose LLMs on classroom writing generation and discipline-specific tasks, while reducing inference energy consumption by 62% and ensuring training data remains strictly within the institutional boundary—thereby mitigating privacy leakage and resource bottlenecks. This study validates the feasibility and superiority of domain-customized small models in resource-constrained, privacy-sensitive environments, offering a deployable and sustainable technical pathway for educational AI.
📝 Abstract
The current trend to improve language model performance seems to be based on scaling up with the number of parameters (e.g. the state of the art GPT4 model has approximately 1.7 trillion parameters) or the amount of training data fed into the model. However this comes at significant costs in terms of computational resources and energy costs that compromise the sustainability of AI solutions, as well as risk relating to privacy and misuse. In this paper we present the Erasmian Language Model (ELM) a small context specific, 900 million parameter model, pre-trained and fine-tuned by and for Erasmus University Rotterdam. We show how the model performs adequately in a classroom context for essay writing, and how it achieves superior performance in subjects that are part of its context. This has implications for a wide range of institutions and organizations, showing that context specific language models may be a viable alternative for resource constrained, privacy sensitive use cases.