Benchmarking Resource-Efficient LLMs for Research Topic Ontology Generation in the Biomedical Field

2026-07-20

AI Summary

This study addresses the inefficiency of manual biomedical ontology construction and the unclear potential of small-scale large language models (LLMs) in identifying complex semantic relationships. It presents the first systematic evaluation of five open-source LLMs with parameter counts ≤9 billion on this task, introducing MeSH-Rel-4K—a novel dataset comprising 4,000 expert-annotated semantic relations from Medical Subject Headings (MeSH). The work compares three adaptation strategies: standard prompting, chain-of-thought prompting, and supervised fine-tuning. Experimental results demonstrate that supervised fine-tuning substantially outperforms prompt-based methods, yielding an average F1 score improvement of 34.1 percentage points. These findings confirm that targeted fine-tuning effectively unlocks the capability of small LLMs for automated, domain-specific ontology construction.

Abstract

Knowledge Organization Systems like Ontologies and taxonomies are fundamental for structuring scientific knowledge, yet their manual curation presents a persistent bottleneck in knowledge management. While Large Language Models (LLMs) offer a scalable mechanism for automated ontology generation, their capacity to classify complex, domain-specific semantics requires systematic evaluation. In this paper, we assess the performance of five small, open-source LLMs (up to 9 billion parameters) in identifying semantic relationships between biomedical concepts. To support this evaluation, we introduce MeSH-Rel-4K, a dataset comprising 4K semantic relationships extracted from the Medical Subject Headings (MeSH). We analyse three adaptation strategies: standard prompting, Chain-of-Thought prompting, and fine-tuning. While parameter-constrained models traditionally struggle with the nuances of in-context logic, our results reveal that targeted fine-tuning increases the average F1-score by 34.1 percentage points. These results confirm that direct fine-tuning effectively exceeds the reasoning bottlenecks of smaller LLMs, providing an accurate, automated methodology for the construction and evolution of specialised biomedical ontologies.

Problem

• ontology generation
• biomedical semantics
• large language models
• knowledge organization
• semantic relationships

Innovation

• resource-efficient LLMs
• ontology generation
• fine-tuning
• MeSH-Rel-4K
• biomedical semantics