Large Language Models as Unified Multimodal Learners for Clinical Prediction

2026-07-16

AI Summary

This work addresses the limitations of conventional clinical prediction systems, which rely on task-specific multimodal fusion architectures that suffer from poor generalizability and high complexity. The authors propose a unified text serialization paradigm that converts both structured and unstructured clinical data into natural language sequences, enabling end-to-end prediction by directly fine-tuning large language models—including ModernBERT, Llama 3.1, Gemma, DeepSeek-R1-Qwen, and Qwen3—without requiring specialized fusion modules. This approach substantially reduces system complexity while achieving performance on par with or superior to existing task-specific multimodal baselines across three clinical prediction tasks. Notably, it outperforms the gradient-boosting model currently used in clinical practice for graft failure prediction, demonstrating strong cross-task and cross-scenario generalization capabilities.

Abstract

Electronic health records combine free-text clinical narratives with structured measurements such as vital signs, laboratory values, and comorbidities. Yet most clinical prediction systems still rely on task-specific fusion architectures, pairing dedicated encoders for each modality with learned combination mechanisms that must be re-engineered for every new task and clinical setting. We propose a simpler alternative: convert all patient data, regardless of modality, into a single natural language sequence and fine-tune a pretrained language model end-to-end, with no architectural modification for fusion. We evaluate this approach across three clinically distinct prediction tasks: in-hospital mortality on MIMIC-III, graft failure prediction using longitudinal data from a German transplant center, and emergency triage classification from ambulance records - comparing encoder-based (ModernBERT) and decoder-based (Llama 3.1, Gemma, DeepSeek-R1-Qwen, Qwen3) fine-tuning against established multimodal baselines and, for graft failure, a gradient boosting model currently used in clinical practice for post-transplant patient management. Across all three tasks, unified textual serialization matches or exceeds task-specific multimodal baselines, and outperforms the clinically deployed gradient boosting system on graft failure prediction. These results indicate that a single serialization-based paradigm, without bespoke fusion architectures, is sufficient for multimodal clinical prediction - substantially reducing system complexity while matching or exceeding specialized designs.

Problem

• clinical prediction
• multimodal fusion
• electronic health records
• task-specific architectures
• structured and unstructured data

Innovation

• unified textual serialization
• multimodal clinical prediction
• large language models
• end-to-end fine-tuning
• modality fusion