BiomedSQL: Text-to-SQL for Scientific Reasoning on Biomedical Knowledge Bases

2025-05-23

AI Summary

Modeling implicit scientific reasoning—such as threshold-based judgments and effect-direction identification—in biomedical text-to-SQL remains challenging. Method: We introduce BMSQL, the first text-to-SQL benchmark explicitly designed for scientific reasoning in biomedicine. It covers realistic scenarios including gene–disease associations, multi-omics causal inference, and drug approval, comprising 68K high-quality triplet-based queries. We construct a BigQuery-integrated biomedical knowledge base and propose a BMSQL agent architecture supporting multi-step reasoning via chain-of-thought prompting and interactive execution verification. Contribution/Results: BMSQL establishes the first evaluation paradigm requiring domain-specific scientific reasoning. Experiments show it improves GPT-4o-mini’s execution accuracy from 59.0% to 62.6%, significantly outperforming existing baselines. The dataset and code are publicly released to advance interpretable, reasoning-aware biomedical AI query systems.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Data Mining & Knowledge Management: Intelligent Query Processing
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

Biomedical researchers increasingly rely on large-scale structured databases for complex analytical tasks. However, current text-to-SQL systems often struggle to map qualitative scientific questions into executable SQL, particularly when implicit domain reasoning is required. We introduce BiomedSQL, the first benchmark explicitly designed to evaluate scientific reasoning in text-to-SQL generation over a real-world biomedical knowledge base. BiomedSQL comprises 68,000 question/SQL query/answer triples grounded in a harmonized BigQuery knowledge base that integrates gene-disease associations, causal inference from omics data, and drug approval records. Each question requires models to infer domain-specific criteria, such as genome-wide significance thresholds, effect directionality, or trial phase filtering, rather than rely on syntactic translation alone. We evaluate a range of open- and closed-source LLMs across prompting strategies and interaction paradigms. Our results reveal a substantial performance gap: GPT-o3-mini achieves 59.0% execution accuracy, while our custom multi-step agent, BMSQL, reaches 62.6%, both well below the expert baseline of 90.0%. BiomedSQL provides a new foundation for advancing text-to-SQL systems capable of supporting scientific discovery through robust reasoning over structured biomedical knowledge bases. Our dataset is publicly available at https://huggingface.co/datasets/NIH-CARD/BiomedSQL, and our code is open-source at https://github.com/NIH-CARD/biomedsql.

Problem

• Mapping qualitative biomedical questions to executable SQL queries
• Handling implicit domain reasoning in text-to-SQL systems
• Evaluating scientific reasoning over biomedical knowledge bases

Innovation

• BiomedSQL benchmark for scientific text-to-SQL
• Multi-step agent BMSQL improves execution accuracy
• Integrates gene-disease associations and omics data