Dynamic Robot-Assisted Surgery with Hierarchical Class-Incremental Semantic Segmentation

2025-08-03

AI Summary

Semantic segmentation models in robot-assisted surgery struggle to continuously adapt to novel anatomical or instrument classes under dynamic surgical scenes while suffering from catastrophic forgetting. Method: We propose TOPICS+, a hierarchical class-incremental semantic segmentation framework. It (1) establishes the first six-task continual learning benchmark for surgical scene segmentation (CISS); (2) introduces a hierarchical pseudo-labeling scheme and customized label taxonomy to extend Syn-Mediverse to 144+ classes; and (3) integrates Dice loss to mitigate class imbalance, jointly optimizing hierarchical cross-task loss and Poincaré-space regularization to enhance representation consistency across tasks. Results: TOPICS+ achieves significant improvements in accuracy and stability across multiple incremental steps and semantic levels, effectively suppressing forgetting. It provides a scalable, robust continual learning framework for real-time scene understanding in dynamic surgical environments.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Segmentation
• Machine Learning: Life-Long and Continual Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Robot-assisted surgeries rely on accurate and real-time scene understanding to safely guide surgical instruments. However, segmentation models trained on static datasets face key limitations when deployed in these dynamic and evolving surgical environments. Class-incremental semantic segmentation (CISS) allows models to continually adapt to new classes while avoiding catastrophic forgetting of prior knowledge, without training on previous data. In this work, we build upon the recently introduced Taxonomy-Oriented Poincaré-regularized Incremental Class Segmentation (TOPICS) approach and propose an enhanced variant, termed TOPICS+, specifically tailored for robust segmentation of surgical scenes. Concretely, we incorporate the Dice loss into the hierarchical loss formulation to handle strong class imbalances, introduce hierarchical pseudo-labeling, and design tailored label taxonomies for robotic surgery environments. We also propose six novel CISS benchmarks designed for robotic surgery environments including multiple incremental steps and several semantic categories to emulate realistic class-incremental settings in surgical environments. In addition, we introduce a refined set of labels with more than 144 classes on the Syn-Mediverse synthetic dataset, hosted online as an evaluation benchmark. We make the code and trained models publicly available at http://topics.cs.uni-freiburg.de.

Problem

• Improving real-time scene understanding in robot-assisted surgeries
• Addressing class imbalance in dynamic surgical environments
• Enhancing segmentation models for evolving surgical scenes

Innovation

• Hierarchical pseudo-labeling for surgical scenes
• Dice loss in hierarchical loss for class imbalance
• Tailored label taxonomies for robotic surgery