A Cost-Effective Multimodal LLM Reasoning Framework for Question Answering over Irregular Clinical Time Series

2026-07-28

AI Summary

This work addresses the challenges of sparsity, asynchronicity, and non-uniform sampling inherent in irregular clinical time series (ICTS) for question-answering tasks by proposing the ClinPRISM framework. ClinPRISM introduces an irregularity-aware multi-scale temporal encoder to capture clinical evidence at multiple granularities and incorporates a temporal evidence distillation mechanism that compresses this information into only 16 large language model (LLM)-compatible tokens. Through a progressive alignment strategy, the framework effectively maps irregular temporal trajectories into the textual embedding space of LLMs. Integrated with instruction tuning and a 4-billion-parameter multimodal LLM, ClinPRISM achieves state-of-the-art performance on standard evaluation benchmarks while maintaining an average inference latency of just 0.15 seconds per query, substantially reducing computational overhead.

Abstract

Question answering (QA) over irregular clinical time series (ICTS) plays a pivotal role in a wide range of healthcare applications. Although recent multimodal time-series large language models (LLMs) have shown considerable promise in general-purpose time-series QA, they remain poorly equipped to model the sparsity, asynchrony, and irregular sampling patterns of clinical observations. To fill this gap, we propose ClinPRISM, a cost-effective multimodal LLM reasoning framework for question answering over ICTS data. First, we devise an irregularity-aware multi-scale encoder to capture sparse clinical evidence at diverse temporal scales. Then, we propose a temporal evidence distiller to integrate representations across these scales and compress them into a small number of LLM-compatible tokens. Moreover, we introduce a progressive alignment strategy that sequentially aligns the irregular trajectories with the LLM's textual embedding space. To facilitate training, we construct 30,000 clinical time series paired with multi-scale descriptions, together with 41,000 instruction-tuning instances spanning 11 tasks. Using a 4-billion-parameter LLM backbone, ClinPRISM achieves state-of-the-art performance on the held-out evaluation benchmark while using only 16 time-series tokens and achieving an average inference latency of 0.15 seconds per question.

Problem

• irregular clinical time series
• question answering
• sparsity
• asynchrony
• irregular sampling

Innovation

• irregular clinical time series
• multimodal LLM
• temporal evidence distillation
• progressive alignment
• cost-effective reasoning