MedLLM: An Open Medical Language Model at the Sub-Billion Scale

2026-07-29

AI Summary

This study addresses the research gap in sub-billion-parameter medical language models by proposing a fully open-source, three-stage training pipeline: general-domain pretraining, domain-specific fine-tuning using a newly curated medical corpus (MedFineWeb), and preference alignment via supervised fine-tuning (SFT) combined with direct preference optimization (DPO). The work reveals a novel task-decoupling phenomenon at this scale, where contextual question answering and factual knowledge recall exhibit divergent performance. Leveraging curriculum learning for sequence scheduling and embedding-similarity-based data filtering, the resulting 0.1B-parameter model achieves contextual QA performance approaching that of 7B-scale medical models and significantly outperforms all ≤7B baselines on the MedMCQA knowledge recall benchmark, demonstrating the high efficiency and task-specific potential of small-scale models in medical applications.

Abstract

Open medical language models have converged on a single scale: every widely used system runs at 7B parameters or more, leaving the sub-billion regime uncharacterized. We present MedLLM, an open 0.1B-parameter medical language model trained through a fully open three-phase pipeline: general pretraining with curriculum sequence-length scheduling, domain fine-tuning on MedFineWeb, a reference-guided medical corpus we release that is selected from general web data by embedding similarity to medical question-answering (QA) data, and preference-aligned fine-tuning combining SFT with direct preference optimization (DPO). Across medical benchmarks, MedLLM shows a pattern visible only at sub-billion scale: medical competence does not degrade uniformly under compression but splits by task type. On context-grounded QA it comes within $2.9$pp of a medically adapted 7B model and surpasses the instruction-tuned and general-purpose 7B baselines; on knowledge-recall QA it stays near the task floor on clinical-vignette MedQA yet significantly exceeds every 7B and sub-7B baseline on MedMCQA, indicating that where recall fails the constraint is model capacity rather than adaptation. This dissociation is masked at 7B, where both capabilities are present, and surfaces only when capacity is scarce.

Problem

• medical language model
• sub-billion scale
• model compression
• task-specific performance
• medical QA

Innovation

• sub-billion language model
• medical domain adaptation
• curriculum pretraining
• reference-guided corpus
• preference-aligned fine-tuning