SPIRIT: Spatio-temporal Pairwise Relational Modeling of Instrument-Tissue Interactions for Surgical Action Triplet Recognition

2026-08-03

AI Summary

This work addresses the limited generalization of surgical action triplet (<instrument, verb, target>) recognition across medical centers by proposing a structured relational modeling paradigm that reframes triplet prediction from flat classification to hierarchical relational reasoning. The approach explicitly models pairwise relationships among instruments, verbs, and targets by first extracting their spatiotemporal representations and then fusing these relations into coherent triplet predictions. To stabilize training, multi-head knowledge distillation is introduced. The study also presents MultiBypass-4C-T40, the first densely annotated dataset designed for cross-center evaluation. Extensive experiments demonstrate that the proposed method significantly outperforms strong baselines under various cross-center evaluation protocols, confirming the effectiveness of explicit relational modeling in enhancing model generalization.

Abstract

Fine-grained understanding of surgical activity is essential for context-aware assistance in the operating room, including safety monitoring, adverse event identification, and skill assessment. Surgical action triplets, defined as tuples of the form <instrument, verb, target>, provide a structured description of instrument-tissue interactions. A key open problem, however, is how to learn triplet representations that remain reliable across institutions, where surgical video varies in acquisition conditions, surgeon style, tool usage, and tissue handling, while existing triplet datasets do not support explicit evaluation of center-wise transfer. To address this problem, we propose \textbf{SPIRIT}, a structured framework for surgical action triplet recognition designed to learn interaction representations that transfer more reliably across centers. Instead of treating each triplet as a flat class label, SPIRIT first learns spatio-temporal representations for instruments, verbs, and targets, then models their pairwise relations, and finally composes them into coherent triplet predictions, with multi-head distillation used to stabilize learning. To evaluate this setting, we establish \textbf{MultiBypass-4C-T40}, a multi-centric dataset for dense surgical action triplet recognition in Roux-en-Y gastric bypass across four geographically distinct centers, with auxiliary phase and step annotations. Across multiple evaluation protocols, SPIRIT consistently outperforms strong recent baselines, highlighting the value of explicit relational reasoning for multi-centric triplet recognition. Code will be available at https://github.com/CAMMA-public/multibypass-4c-t40.

Problem

• surgical action triplet
• cross-center generalization
• instrument-tissue interaction
• multi-centric transfer
• structured representation

Innovation

• surgical action triplet
• spatio-temporal modeling
• pairwise relational reasoning
• multi-centric transfer
• multi-head distillation