Medical Vision Language Models as Policies for Robotic Surgery

2025-10-07

AI Summary

Vision-based proximal policy optimization (PPO) for laparoscopic surgery faces challenges including high-dimensional image inputs, sparse reward signals, and difficulty in extracting task-relevant features. Method: This work introduces MedFlamingo—a medical-domain pretrained vision-language model (VLM)—into the PPO policy network for the first time. Leveraging its medical prior knowledge, MedFlamingo generates high-level planning tokens, enabling end-to-end learning from raw laparoscopic video streams and textual instructions within the LapGym simulator. Contribution/Results: Evaluated on five laparoscopic tasks, the approach achieves an average success rate exceeding 70%, outperforming baseline methods by 66.7%–1114.3% and significantly accelerating policy convergence. This study pioneers the integration of domain-specific VLMs into robotic surgical reinforcement learning, effectively bridging low-level perception and high-level semantic decision-making.

Technology Category

• Natural Language Processing: Safety and Robustness
• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Vision-based Proximal Policy Optimization (PPO) struggles with visual observation-based robotic laparoscopic surgical tasks due to the high-dimensional nature of visual input, the sparsity of rewards in surgical environments, and the difficulty of extracting task-relevant features from raw visual data. We introduce a simple approach integrating MedFlamingo, a medical domain-specific Vision-Language Model, with PPO. Our method is evaluated on five diverse laparoscopic surgery task environments in LapGym, using only endoscopic visual observations. MedFlamingo PPO outperforms and converges faster compared to both standard vision-based PPO and OpenFlamingo PPO baselines, achieving task success rates exceeding 70% across all environments, with improvements ranging from 66.67% to 1114.29% compared to baseline. By processing task observations and instructions once per episode to generate high-level planning tokens, our method efficiently combines medical expertise with real-time visual feedback. Our results highlight the value of specialized medical knowledge in robotic surgical planning and decision-making.

Problem

• Addresses high-dimensional visual input challenges in robotic surgery
• Solves sparse reward issues in surgical task environments
• Overcomes difficulty extracting task-relevant features from visual data

Innovation

• Integrates medical vision-language model with reinforcement learning
• Generates planning tokens from observations and instructions
• Combines medical expertise with real-time visual feedback