RoboSurg-VQA: A Multimodal Benchmark for Surgical Segmentation-Aware Visual Question Answering

2026-05-21

AI Summary

This work addresses the limitation of relying solely on segmentation masks for clinical semantic understanding in robot-assisted minimally invasive surgery by introducing RoboSurg-VQA, the first surgical visual question answering benchmark that integrates segmentation-aware and clinical semantics. Built upon existing surgical segmentation datasets, the benchmark leverages a constrained language model to generate candidate question-answer pairs, which are then refined through automated consistency checks and expert human review. The resulting large-scale, structured VQA dataset encompasses multiple dimensions—including anatomical structures, surgical instruments, artifacts, field-of-view quality, and procedural context—enabling robust evaluation under challenging intraoperative conditions such as occlusion, smoke, and bleeding. The study further provides baseline model performance and in-depth challenge analysis, advancing multimodal semantic understanding in surgical environments.

Abstract

Reliable visual understanding in robot-assisted and minimally invasive surgery (RMIS/MIS) demands more than accurate masks: in clinical practice, clinicians pose language-like questions about procedural context, visibility, artefacts, and the presence of anatomical structures and surgical instruments, often under degraded views caused by occlusion, smoke, bleeding, and specular highlights. We present \textbf{RoboSurg-VQA}, a segmentation-aware visual question answering (VQA) benchmark built by repurposing public surgical segmentation datasets under a shared schema. Each frame is paired with a fixed set of clinically motivated questions spanning procedure context, anatomy (including region), imaging modality/view, surgical artefacts, image quality, and basic visibility and spatial attributes, with closed answer sets to enable consistent evaluation. To scale annotation, we generate candidate answers via constrained prompting with automatic validity and consistency checks, followed by human auditing to improve plausibility and label consistency. We report benchmark statistics, sanity baselines, and common evaluation challenges under challenging surgical conditions. The code will be available on https://github.com/ziyangwang007/Robosurg-VQA.

Problem

• visual question answering
• surgical segmentation
• robot-assisted surgery
• multimodal understanding
• clinical reasoning

Innovation

• segmentation-aware VQA
• surgical visual question answering
• multimodal benchmark
• constrained prompting
• clinical language grounding