SurgiQ: A Large-Scale Multi-Domain Benchmark for Evaluating Surgical Understanding in Large Language Models

2026-06-06

AI Summary

Existing medical benchmarks inadequately assess critical capabilities of large language models in surgical domains, such as procedural reasoning, decision trade-offs, and understanding negation. To address this gap, this work introduces SurgiQ, the first multidimensional benchmark for surgical comprehension, comprising 13,055 four-option multiple-choice questions spanning six surgical specialties and four question types. Questions are derived from textbooks, examination materials, and open-access literature, and are rigorously constructed through a multi-stage generation pipeline followed by expert review and quality control. Using a unified log-likelihood evaluation protocol, the benchmark is applied to 35 open-source models. Results show that general-purpose models (e.g., Qwen2.5) outperform most biomedical-specialized counterparts, achieving a peak accuracy of 68.1%—significantly above the 25% random baseline—yet still exhibit high-confidence errors on clinically plausible distractors, underscoring the unique challenges of surgical understanding.

Abstract

Reliable evaluation of large language models in surgery remains underdeveloped. Broad medical benchmarks test clinical knowledge, while surgery requires procedural reasoning, management trade-offs, negation handling, and selection among plausible operative decisions. We present SurgiQ, a text-only, source-grounded benchmark of 13,055 four-option multiple-choice questions spanning six surgical domains and four question formats: case-based, reasoning, best-option, and negative. SurgiQ is constructed from surgical textbooks, open-access papers, and examination material using a multi-stage generation, verification, and expert-audit pipeline. We evaluate 35 open-weight LLMs under a unified log-likelihood protocol. Our results show substantial remaining headroom: smaller models often remain near the 25\% random baseline, while the best model reaches 68.1\% accuracy. General-purpose models, especially Qwen2.5, outperform most biomedical models, suggesting that current medical specialization does not yet provide sufficiently broad surgical coverage. Calibration and error analysis further show that even strong models make confident mistakes on clinically plausible distractors, motivating more reliable and broader surgical LLM evaluation.

Problem

• surgical understanding
• large language models
• evaluation benchmark
• procedural reasoning
• clinical decision-making

Innovation

• surgical understanding
• large language models
• benchmark evaluation
• procedural reasoning
• expert-audited dataset