SurgNarrator: A Generative Retrieval Framework for Surgical Video Understanding

2026-08-05

AI Summary

Existing methods for surgical video understanding struggle to simultaneously achieve real-time performance and complex semantic parsing, failing to meet the clinical dual demands of low latency and high accuracy. This work introduces generative retrieval into the surgical domain for the first time, proposing a surgery-oriented generative retrieval framework that defines a clinical semantic space through a surgery-specific vocabulary. By integrating temporal-aware contrastive learning with a procedure-aware hierarchical retrieval mechanism, the framework enables efficient and precise semantic interpretation. Built upon the Qwen3-VL-Embedding-8B model and leveraging surgical captions to construct a domain-specific lexicon, the approach surpasses current state-of-the-art methods in zero-shot performance across twelve benchmarks while reducing inference latency by over two orders of magnitude compared to generative baselines.

Abstract

Surgical procedures unfold as structured and recurring clinical events, whose real-time understanding via intraoperative surgical videos is critical for intraoperative decision-making and support. However, existing video understanding methods force a trade-off: autoregressive video-language models support comprehensive reasoning but are not practical for time-sensitive clinical applications, whereas contrastive models offer low latency but struggle with complex scene understanding. Recently, generative retrieval has been explored for general-domain video understanding, but transferring it to surgery is not trivial because near-identical visual appearances may indicate semantically distinct events, and the terminology involved is highly surgery-specific. To this end, we propose SurgNarrator, a new generative retrieval framework tailored for surgical video understanding. We construct a well-curated surgery-centric vocabulary from surgical captions to define a clinically meaningful retrieval space. We then adapt the pre-trained Qwen3-VL-Embedding-8B to learn discriminative clinical representations with a temporally-aware contrastive objective. During inference, a hierarchical, procedure-aware retrieval strategy narrows the search space to the relevant procedure type, delivering fast and effective responses. Our method is comprehensively evaluated on twelve benchmarks in a zero-shot setting and achieves consistent performance gains over state-of-the-art baselines, while reducing output-stage latency by more than two orders of magnitude compared with the generative baseline.

Problem

• surgical video understanding
• generative retrieval
• real-time clinical decision-making
• surgery-specific terminology
• video-language models

Innovation

• generative retrieval
• surgical video understanding
• temporally-aware contrastive learning
• surgery-specific vocabulary
• hierarchical retrieval