LAVIFT: Latent-Action-Guided Vision Fine-Tuning for Surgical Interaction Recognition

2026-07-22

AI Summary

This work addresses the limitations of current vision-language models in fine-grained surgical instrument-tissue interaction recognition, particularly their weak spatial localization and tendency to learn features misaligned with action-relevant regions. To overcome these issues, we propose the first end-to-end latent action-guided fine-tuning framework that integrates latent action signals into vision-language adaptation. Our approach leverages an inverse dynamics model to capture visual changes induced by actions, employs a forward world model to guide the encoder’s attention toward action-related regions, and introduces unsupervised block-level regularization to prevent local feature collapse. Requiring neither bounding boxes nor pseudo-labels, the method significantly improves interaction recognition accuracy and vision-language alignment across multiple encoders and datasets. Representation analysis confirms enhanced spatial localization and semantic consistency within interaction regions.

Abstract

Understanding instrument-tissue interactions is essential for context-aware surgical AI and autonomous robotic surgery. Pretrained vision-language models (VLMs) and vision encoders offer an alternative to conventional interaction classifiers by transferring broad visual and semantic knowledge. However, adapting them to fine-grained surgical interactions remains challenging: (1) freezing the vision encoder depends entirely on pretrained representations that may retain noise and provide weak spatial localization, while (2) full fine-tuning can improve global semantic alignment without ensuring that the encoder learns meaningful features in the correct action region. We address these limitations by introducing LAViFiT, an end-to-end latent-action-guided framework for vision-language fine-tuning. An inverse dynamics model captures the visual changes induced by each action, while a forward world model drives the encoder to represent action-relevant regions. A patch-level SIG Regularizer further prevents local feature collapse without additional supervision, such as bounding boxes or pseudo-labels. Experiments across multiple encoders and datasets improve recognition and image-text alignment, while representation analyses show stronger grounding over the complete instrument-tissue interaction region and more spatially coherent features.

Problem

• surgical interaction recognition
• vision-language models
• fine-tuning
• spatial localization
• instrument-tissue interaction

Innovation

• latent-action guidance
• vision-language fine-tuning
• inverse dynamics model
• forward world model
• SIG regularizer