Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Review

2025-02-16

AI Summary

This paper addresses the bottleneck in endoscopic video understanding during minimally invasive surgery (MIS). It systematically reviews foundational models—including Vision Transformers (ViTs) and the Segment Anything Model (SAM)—applied to intraoperative instrument segmentation, tracking, and surgical phase recognition. Adopting a “paradigm shift” perspective, it presents the first comprehensive taxonomy of foundational-model-driven evolution in surgical visual understanding. Results indicate substantial improvements in segmentation accuracy and phase classification performance; however, critical challenges persist, including data heterogeneity across surgical settings, real-time inference latency, and barriers to clinical integration. The core contributions are threefold: (1) enhanced clinical adaptability through domain-aware fine-tuning and interpretable interfaces; (2) ethically aligned model design incorporating surgeon-in-the-loop validation and bias mitigation; and (3) construction of dynamic generalization capabilities for cross-procedure, cross-device robustness. Finally, the work proposes an actionable, technology–clinical co-development roadmap to accelerate safe, scalable deployment of AI in MIS.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Deep Neural Architectures and Foundation Models
• Cognitive Modeling & Cognitive Systems: Other Foundations of Cognitive Modeling & Systems

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Recent advancements in machine learning (ML) and deep learning (DL), particularly through the introduction of foundational models (FMs), have significantly enhanced surgical scene understanding within minimally invasive surgery (MIS). This paper surveys the integration of state-of-the-art ML and DL technologies, including Convolutional Neural Networks (CNNs), Vision Transformers (ViTs), and foundational models like the Segment Anything Model (SAM), into surgical workflows. These technologies improve segmentation accuracy, instrument tracking, and phase recognition in surgical endoscopic video analysis. The paper explores the challenges these technologies face, such as data variability and computational demands, and discusses ethical considerations and integration hurdles in clinical settings. Highlighting the roles of FMs, we bridge the technological capabilities with clinical needs and outline future research directions to enhance the adaptability, efficiency, and ethical alignment of AI applications in surgery. Our findings suggest that substantial progress has been made; however, more focused efforts are required to achieve seamless integration of these technologies into clinical workflows, ensuring they complement surgical practice by enhancing precision, reducing risks, and optimizing patient outcomes.

Problem

• Enhancing surgical scene understanding
• Integrating AI in surgical workflows
• Improving segmentation and instrument tracking

Innovation

• Foundational Models enhance surgical understanding
• CNNs and ViTs improve surgical video analysis
• AI integration faces clinical and ethical challenges