OntoBook: Ontology-Grounded Synthetic Textbooks for Medical Encoder Pretraining

2026-07-21

AI Summary

This study addresses the scarcity of high-quality, structure-aware pretraining data for medical coding by proposing a novel approach that leverages medical ontology graph structures as natural language pretraining signals. The authors generate structured paths via random walks over the ontology and employ a large language model to reconstruct them into textbook-style synthetic texts. They then jointly pretrain ModernCamemBERT—a French medical encoder—using masked language modeling and relation prediction objectives, ensuring cross-task alignment through this dual-objective framework. Experimental results demonstrate substantial improvements over MLM-only baselines on three French medical coding benchmarks: FRACCO, Cantemist-FR, and Distemist-FR, with micro F1 gains of +2.5 and +8.0, respectively. The project publicly releases 1.3 million synthetic medical textbook passages and model checkpoints.

Abstract

We present OntoBook, a method that converts medical ontology structure into pretraining signal for encoder language models. Our approach has three stages: random walks through ontology graphs capture hierarchical and causal relations between medical codes, a large language model reformulates these walks into fluent textbook-style prose, and the resulting text is used to train ModernCamemBERT, a 149M-parameter French encoder, with two objectives on the same data: masked language modeling and relation prediction between code pairs. On three French medical coding benchmarks (FRACCO, Cantemist-FR, Distemist-FR), OntoBook achieves significant improvements over MLM-only pretraining, with +2.5 micro-F1 on FRACCO and +8.0 micro-F1 on Distemist. We find that alignment between objectives is necessary: misaligned training, where each task uses different data, causes a 30-point degradation. We release 1.3 million LLM-reformulated medical textbooks across three French ontologies (CIM-10, CCAM, ATC) and pretrained model checkpoints.

Problem

• medical ontology
• encoder pretraining
• medical coding
• synthetic textbooks
• language models

Innovation

• ontology-grounded pretraining
• synthetic textbook generation
• medical encoder pretraining
• relation prediction
• masked language modeling