TEMSET-24K: Densely Annotated Dataset for Indexing Multipart Endoscopic Videos using Surgical Timeline Segmentation

2025-02-10

AI Summary

Endoscopic surgical video indexing has long relied on inefficient manual annotation, while the scarcity of publicly available, densely annotated datasets severely hinders the development of automated methods. Method: We introduce TEMS, the first public 24K-scale dataset of fine-grained endoscopic surgical video snippets, annotated using a hierarchical “phase–task–action” triadic schema co-designed with clinical experts to enable precise, interpretable, and openly shareable surgical workflow modeling. We propose STALNet, an end-to-end framework integrating ConvNeXt, ViT, and Swin V2 encoders to jointly perform surgical phase segmentation and multi-granularity temporal localization. Contribution/Results: STALNet achieves 0.99 accuracy and F1 score on critical phases (e.g., Setup, Suturing), substantially advancing automated surgical video indexing. TEMS fills a critical gap in high-quality, expert-validated annotations and establishes a new benchmark and methodological paradigm for surgical video understanding.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

Indexing endoscopic surgical videos is vital in surgical data science, forming the basis for systematic retrospective analysis and clinical performance evaluation. Despite its significance, current video analytics rely on manual indexing, a time-consuming process. Advances in computer vision, particularly deep learning, offer automation potential, yet progress is limited by the lack of publicly available, densely annotated surgical datasets. To address this, we present TEMSET-24K, an open-source dataset comprising 24,306 trans-anal endoscopic microsurgery (TEMS) video micro-clips. Each clip is meticulously annotated by clinical experts using a novel hierarchical labeling taxonomy encompassing phase, task, and action triplets, capturing intricate surgical workflows. To validate this dataset, we benchmarked deep learning models, including transformer-based architectures. Our in silico evaluation demonstrates high accuracy (up to 0.99) and F1 scores (up to 0.99) for key phases like Setup and Suturing. The STALNet model, tested with ConvNeXt, ViT, and SWIN V2 encoders, consistently segmented well-represented phases. TEMSET-24K provides a critical benchmark, propelling state-of-the-art solutions in surgical data science.

Problem

• Automated indexing of endoscopic surgical videos
• Lack of densely annotated surgical datasets
• Validation of deep learning models for surgical workflow segmentation

Innovation

• Deep learning for video indexing
• Transformer-based surgical segmentation
• Hierarchical labeling taxonomy application