Real-Time Visual Obstruction Detection in Surgical Augmented Reality

2026-07-31

AI Summary

In surgical augmented reality, virtual content can occlude critical instruments, impairing intraoperative perception. To address this challenge, this work proposes a low-latency visual occlusion detection method that integrates a cascaded vision-language model with image segmentation. The approach introduces a segmentation-guided early-exit mechanism and an attention-based visual token pruning strategy to significantly enhance inference efficiency while preserving accuracy. Evaluated on a custom pseudo-AR dataset, the method achieves a detection accuracy of 87.43% with an end-to-end latency of only 479 milliseconds—representing a 62.90% reduction compared to a cloud-based large-model baseline—thus effectively balancing real-time performance and precision.

Abstract

Surgical augmented reality (AR) can provide contextual guidance by overlaying virtual annotations, tool cues, and procedural information onto the surgical workspace. However, the virtual content may obstruct task-relevant real-world information, such as surgical instruments, and interfere with users' perception during time-sensitive surgical tasks. In this paper, we investigate visual obstruction detection for surgical AR and present a latency-aware pipeline that combines vision-language model (VLM)-based surgical-object recognition with segmentation-based obstruction reasoning. To reduce inference overhead, the system adopts a cascaded small-to-large VLM architecture with segmentation-guided early exiting and attention-based visual token pruning. The small VLM handles easy frames when its key-object prediction is supported by segmentation consistency, while difficult frames are forwarded to a large VLM with pruned visual tokens. We construct a pseudo-AR surgical obstruction detection benchmark by overlaying virtual content onto surgical-tool images and labeling whether the virtual content obstructs task-relevant instruments. Evaluation results show that the proposed system achieves 87.43% obstruction detection accuracy with an average end-to-end latency of 479 ms, reducing latency by 62.90% compared with a cloud large-model baseline. These results demonstrate the feasibility of latency-aware obstruction detection for surgical AR and motivate future work on dynamic surgical videos, multi-object scenes, and clinically grounded AR guidance content.

Problem

• surgical augmented reality
• visual obstruction
• real-time detection
• surgical instruments
• perceptual interference

Innovation

• vision-language model
• real-time obstruction detection
• latency-aware pipeline
• visual token pruning
• surgical augmented reality