Tracking Everything in Robotic-Assisted Surgery

2024-09-29

arXiv.org

AI Summary

In robotic-assisted minimally invasive surgery (RAMIS), dense, long-term tracking of soft tissue and surgical instruments is prone to drift under rapid motion, severe occlusion, and motion blur. To address this, we propose SurgMotion—a novel tracking framework built upon the Tracking Any Point (TAP) paradigm. SurgMotion integrates surgical motion priors, occlusion-aware feature matching, and multi-frame temporal refinement, augmented by a structural consistency constraint to enhance robustness. As a key contribution, we introduce the first annotated video benchmark dataset specifically designed for surgical visual tracking. Extensive experiments demonstrate that SurgMotion reduces average instrument tracking error by 37% on our benchmark compared to state-of-the-art TAP-based methods, achieving superior accuracy and stability—particularly in highly dynamic and heavily occluded scenarios.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: Motion and Path Planning
• Planning, Routing, and Scheduling: Activity and Plan Recognition

Application Category

• Security and Privacy: Tracking, profiling, and countermeasures against them
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Accurate tracking of tissues and instruments in videos is crucial for Robotic-Assisted Minimally Invasive Surgery (RAMIS), as it enables the robot to comprehend the surgical scene with precise locations and interactions of tissues and tools. Traditional keypoint-based sparse tracking is limited by featured points, while flow-based dense two-view matching suffers from long-term drifts. Recently, the Tracking Any Point (TAP) algorithm was proposed to overcome these limitations and achieve dense accurate long-term tracking. However, its efficacy in surgical scenarios remains untested, largely due to the lack of a comprehensive surgical tracking dataset for evaluation. To address this gap, we introduce a new annotated surgical tracking dataset for benchmarking tracking methods for surgical scenarios, comprising real-world surgical videos with complex tissue and instrument motions. We extensively evaluate state-of-the-art (SOTA) TAP-based algorithms on this dataset and reveal their limitations in challenging surgical scenarios, including fast instrument motion, severe occlusions, and motion blur, etc. Furthermore, we propose a new tracking method, namely SurgMotion, to solve the challenges and further improve the tracking performance. Our proposed method outperforms most TAP-based algorithms in surgical instruments tracking, and especially demonstrates significant improvements over baselines in challenging medical videos. Our code and dataset are available at https://github.com/zhanbh1019/SurgicalMotion.

Problem

• Evaluate TAP algorithm's efficacy in surgical scenarios
• Address lack of surgical tracking dataset for evaluation
• Improve tracking performance in challenging surgical conditions

Innovation

• Introduces new annotated surgical tracking dataset
• Proposes SurgMotion for improved tracking performance
• Evaluates TAP-based algorithms on surgical scenarios