🤖 AI Summary
Current AI-driven healthcare models predominantly rely on electronic health records (EHRs), overlooking real-time, multimodal patient–clinician interactions—such as audio, video, and text—which constitute a core clinical information source and risk reinforcing a narrow biomedical perspective.
Method: We developed and validated the first reproducible, longitudinal, multimodal system that systematically integrates 360-degree audiovisual recordings, structured patient-reported outcome (PRO) questionnaires, and EHR data to construct a standardized dataset dynamically reflecting authentic clinical encounters. Our methodology encompasses ethically compliant, cross-modal data acquisition, linkage, and governance workflows.
Contribution/Results: As of August 2025, the system achieved 97% clinician and 75% patient participation rates; 76% of visits were fully multimodally recorded, and 96% of follow-up PRO questionnaires were returned—demonstrating high feasibility and implementation efficacy. This infrastructure overcomes the limitations of unimodal EHRs and establishes a new paradigm and foundational resource for interaction-centered AI in healthcare.
📝 Abstract
The promise of AI in medicine depends on learning from data that reflect what matters to patients and clinicians. Most existing models are trained on electronic health records (EHRs), which capture biological measures but rarely patient-clinician interactions. These relationships, central to care, unfold across voice, text, and video, yet remain absent from datasets. As a result, AI systems trained solely on EHRs risk perpetuating a narrow biomedical view of medicine and overlooking the lived exchanges that define clinical encounters. Our objective is to design, implement, and evaluate the feasibility of a longitudinal, multimodal system for capturing patient-clinician encounters, linking 360 degree video/audio recordings with surveys and EHR data to create a dataset for AI research. This single site study is in an academic outpatient endocrinology clinic at Mayo Clinic. Adult patients with in-person visits to participating clinicians are invited to enroll. Encounters are recorded with a 360 degree video camera. After each visit, patients complete a survey on empathy, satisfaction, pace, and treatment burden. Demographic and clinical data are extracted from the EHR. Feasibility is assessed using five endpoints: clinician consent, patient consent, recording success, survey completion, and data linkage across modalities. Recruitment began in January 2025. By August 2025, 35 of 36 eligible clinicians (97%) and 212 of 281 approached patients (75%) had consented. Of consented encounters, 162 (76%) had complete recordings and 204 (96%) completed the survey. This study aims to demonstrate the feasibility of a replicable framework for capturing the multimodal dynamics of patient-clinician encounters. By detailing workflows, endpoints, and ethical safeguards, it provides a template for longitudinal datasets and lays the foundation for AI models that incorporate the complexity of care.