Composable Strategy Framework with Integrated Video-Text based Large Language Models for Heart Failure Assessment

2025-02-23

AI Summary

Heart failure (HF) presents significant challenges in prognosis assessment and treatment optimization due to its multifactorial etiology and complex interplay among clinical factors. To address this, we propose the first composable multimodal framework specifically designed for HF diagnosis and management. Inspired by clinical physician–patient consultation workflows, the framework integrates heterogeneous data—including video (e.g., physical sign observation), textual clinical notes/reports, physical examination findings, and structured medical history—via video–text joint modeling, clinical-knowledge-guided cross-modal alignment, dynamic policy orchestration, and end-to-end reasoning with multimodal large language models (MLLMs). A key innovation is its support for interpretable, pathology-informed attribution analysis of prognostic biomarkers. In HF prognosis prediction, our method achieves a 12.7% accuracy improvement over the best unimodal baseline and, for the first time, enables quantitative modeling of associations between critical biomarkers and clinical outcomes.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

Heart failure is one of the leading causes of death worldwide, with millons of deaths each year, according to data from the World Health Organization (WHO) and other public health agencies. While significant progress has been made in the field of heart failure, leading to improved survival rates and improvement of ejection fraction, there remains substantial unmet needs, due to the complexity and multifactorial characteristics. Therefore, we propose a composable strategy framework for assessment and treatment optimization in heart failure. This framework simulates the doctor-patient consultation process and leverages multi-modal algorithms to analyze a range of data, including video, physical examination, text results as well as medical history. By integrating these various data sources, our framework offers a more holistic evaluation and optimized treatment plan for patients. Our results demonstrate that this multi-modal approach outperforms single-modal artificial intelligence (AI) algorithms in terms of accuracy in heart failure (HF) prognosis prediction. Through this method, we can further evaluate the impact of various pathological indicators on HF prognosis,providing a more comprehensive evaluation.

Problem

• Develops a multi-modal AI framework
• Enhances heart failure prognosis accuracy
• Integrates video-text data for holistic assessment

Innovation

• Composable strategy framework
• Integrated video-text analysis
• Multi-modal AI algorithms