Representation-guided in-context learning for medical image interpretation with multimodal large language models

2026-09-20

AI Summary

为解决医疗图像解读中多模态大语言模型适应性问题，提出无需训练的表示引导上下文学习方法，通过检索对齐案例提高分类和视觉问答性能。

Abstract

Medical image interpretation is central to diagnosis and care, yet adapting general-purpose multimodal large language models (MLLMs) often requires resource-intensive domain-specific fine-tuning. Here we introduce representation-guided in-context learning (RG-ICL), a training-free inference framework that retrieves query-aligned demonstrations using frozen encoders, without task-specific parameter updates. Across eight datasets spanning histopathology, radiology and retinal fundoscopy, RG-ICL improved classification (mean gain 20 percentage points) and visual question answering (VQA) (mean gain 13 percentage points) over no-context and conventional ICL, approaching or exceeding training-based comparators. Which cases were retrieved mattered more than how many: 6 query-aligned cases outperformed up to 32 randomly selected ones, whereas fixed or random cases often reduced accuracy below baseline. For VQA, aligning reference cases with both image content and question intent produced further gains. These findings indicate that for medical image interpretation, curating which reference cases an MLLM sees is a practical alternative to retraining it.

Problem

• medical image interpretation
• multimodal large language models
• domain-specific fine-tuning

Innovation

• representation-guided in-context learning
• frozen encoders
• query-aligned demonstrations
• medical image interpretation
• multimodal large language models