Automated Report-Derived Oncology VQA Benchmark for Evaluating Vision-Language Models on 3D Medical Imaging

2026-06-01

AI Summary

Current evaluation benchmarks for medical vision-language models (VLMs) are limited by small scale, reliance on manual annotation, and contamination from pretraining data, hindering an accurate assessment of visual understanding capabilities. This work proposes an automated, agent-driven framework for constructing medical visual question answering (VQA) benchmarks by integrating clinical reporting templates (RADS-style questions), large language model–based generation and verification, 3D medical image processing, and rigorous data contamination controls. The approach enables the creation of high-quality multiple-choice VQA datasets from private radiology reports and 3D tumor imaging without per-question human annotation. Zero-shot evaluations across four cancer cohorts reveal no single VLM consistently outperforms others; notably, lung CT questions are often answerable from text alone, whereas liver tasks critically depend on visual input, highlighting fundamental differences in image reliance across cancer types.

Abstract

Evaluating vision-language models (VLMs) on medical images requires benchmarks that are clinically grounded, scalable, and controlled for evaluation confounds. Existing public benchmarks are limited in scale, manually annotated, or potentially leaked into VLM pretraining corpora. We present an automated agent-driven pipeline that generates multiple-choice VQA datasets directly from paired private radiology reports and 3D oncology imaging, producing two complementary question types: RADS-style questions deterministically derived from clinician-defined reporting schemas, and radiology report-derived questions generated by an LLM from radiologist findings and verified against the source report. Applied to four in-house cancer cohorts, the pipeline yields an instance-contamination-controlled benchmark without per-question human annotation. Zero-shot evaluation of six VLMs reveals no dominant model and substantial headroom across all cells. A blind ablation reveals that visual reliance is highly dataset-specific: liver Report-derived questions genuinely require the image, while Lung CT is essentially solvable without it - the leading closed model exceeds its sighted accuracy on Lung CT when blinded - indicating that even private clinical data does not guarantee a contamination-controlled read of visual capability. The pipeline is released as an open agent skill for in-house redeployment.

Problem

• vision-language models
• medical imaging
• VQA benchmark
• evaluation confounds
• 3D oncology

Innovation

• automated VQA benchmark
• vision-language models
• 3D medical imaging
• agent-driven pipeline
• contamination-controlled evaluation