Integrating Virtual Reality and Large Language Models for Team-Based Non-Technical Skills Training and Evaluation in the Operating Room

2026-01-19

AI Summary

This study addresses the lack of scalable, structured training and objective assessment tools for non-technical skills—such as communication and teamwork—in laparoscopic surgical emergencies. To bridge this gap, the authors propose VORTeX, an immersive platform integrating multi-user virtual reality with large language models (LLMs). VORTeX uniquely combines structured prompt engineering grounded in the Non-Technical Skills for Surgeons (NOTSS) framework with dynamic interaction graph generation to enable automated, interpretable evaluation of team-based non-technical performance. In a pilot study involving twelve surgical professionals, the system demonstrated high levels of immersion and intuitiveness. The LLM successfully reconstructed communication networks reflective of real-world surgical hierarchies, thereby validating both the assessment efficacy of the approach and its potential for privacy-compliant distributed training.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Multiagent Systems: Teamwork

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Although effective teamwork and communication are critical to surgical safety, structured training for non-technical skills (NTS) remains limited compared with technical simulation. The ACS/APDS Phase III Team-Based Skills Curriculum calls for scalable tools that both teach and objectively assess these competencies during laparoscopic emergencies. We introduce the Virtual Operating Room Team Experience (VORTeX), a multi-user virtual reality (VR) platform that integrates immersive team simulation with large language model (LLM) analytics to train and evaluate communication, decision-making, teamwork, and leadership. Team dialogue is analyzed using structured prompts derived from the Non-Technical Skills for Surgeons (NOTSS) framework, enabling automated classification of behaviors and generation of directed interaction graphs that quantify communication structure and hierarchy. Two laparoscopic emergency scenarios, pneumothorax and intra-abdominal bleeding, were implemented to elicit realistic stress and collaboration. Twelve surgical professionals completed pilot sessions at the 2024 SAGES conference, rating VORTeX as intuitive, immersive, and valuable for developing teamwork and communication. The LLM consistently produced interpretable communication networks reflecting expected operative hierarchies, with surgeons as central integrators, nurses as initiators, and anesthesiologists as balanced intermediaries. By integrating immersive VR with LLM-driven behavioral analytics, VORTeX provides a scalable, privacy-compliant framework for objective assessment and automated, data-informed debriefing across distributed training environments.

Problem

• non-technical skills
• team-based training
• surgical safety
• objective assessment
• operating room

Innovation

• Virtual Reality
• Large Language Models
• Non-Technical Skills
• Team-Based Training
• Behavioral Analytics