CGBench: Benchmarking Language Model Scientific Reasoning for Clinical Genetics Research

2025-10-13

AI Summary

In clinical genetics, variant interpretation remains heavily manual and inefficient; existing LLM evaluation benchmarks are narrow in scope and disconnected from real-world research needs. Method: We introduce CGBench—the first scientific reasoning benchmark for clinical genetics—built upon expert-curated ClinGen annotations. It features multi-level literature interpretation tasks: fine-grained experimental evidence extraction, evidence strength assessment, and conclusion classification. Our evaluation framework integrates structured extraction protocols, an evidence grading schema, and an LM-judge–driven automatic explanation quality assessment. We evaluate eight state-of-the-art LLMs. Contribution/Results: Reasoning-optimized models excel at fine-grained extraction, while non-reasoning models perform better on high-level classification. Critically, all models exhibit pervasive hallucination and misinterpretation—even when classifications are correct, explanations frequently contain fabricated content. CGBench establishes a reproducible, scalable evaluation paradigm for AI in precision medicine.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Variant and gene interpretation are fundamental to personalized medicine and translational biomedicine. However, traditional approaches are manual and labor-intensive. Generative language models (LMs) can facilitate this process, accelerating the translation of fundamental research into clinically-actionable insights. While existing benchmarks have attempted to quantify the capabilities of LMs for interpreting scientific data, these studies focus on narrow tasks that do not translate to real-world research. To meet these challenges, we introduce CGBench, a robust benchmark that tests reasoning capabilities of LMs on scientific publications. CGBench is built from ClinGen, a resource of expert-curated literature interpretations in clinical genetics. CGBench measures the ability to 1) extract relevant experimental results following precise protocols and guidelines, 2) judge the strength of evidence, and 3) categorize and describe the relevant outcome of experiments. We test 8 different LMs and find that while models show promise, substantial gaps exist in literature interpretation, especially on fine-grained instructions. Reasoning models excel in fine-grained tasks but non-reasoning models are better at high-level interpretations. Finally, we measure LM explanations against human explanations with an LM judge approach, revealing that models often hallucinate or misinterpret results even when correctly classifying evidence. CGBench reveals strengths and weaknesses of LMs for precise interpretation of scientific publications, opening avenues for future research in AI for clinical genetics and science more broadly.

Problem

• Evaluating language models' scientific reasoning in clinical genetics research
• Benchmarking LM capabilities for precise experimental data extraction
• Assessing LM performance on evidence interpretation from scientific literature

Innovation

• Benchmark tests reasoning on clinical genetics publications
• Measures evidence extraction and strength judgment capabilities
• Evaluates models using expert-curated literature interpretations