Frozen Models, Evolving Expertise: Model-Agnostic Learning from Deployment Experience for Multimodal Medical AI

2026-10-06

AI Summary

This study addresses the limitations of frozen multimodal medical large models, including their inability to learn continuously, high fine-tuning costs, and susceptibility to overfitting in parameter-free methods. To overcome these challenges, we propose a model-agnostic, training-free online evolution framework that integrates inference skill guidance with structured knowledge memory. Specifically, this work introduces the first multimodal external knowledge base fusing textual facts and visual exemplars, enabling frozen foundation models to accumulate clinical expertise without weight updates. Furthermore, a dynamic update strategy validated on novel cases is designed to mitigate overfitting. Experiments demonstrate that the proposed framework achieves performance gains of up to 34.2% across six medical benchmarks, exhibits strong cross-model transferability, and successfully generalizes to non-medical visual reasoning tasks.

Abstract

Large language models (LLMs) and vision-language models (VLMs) are usually frozen after deployment, so they do not learn from the cases they solve. This is especially concerning in medicine, where new clinical evidence, updated guidelines, and new therapies can change established practice. Fine-tuning can update the model, but it requires access to model weights and additional training. Parameter-free methods avoid training, but they may overfit a fixed validation set, lack reliable domain knowledge, or lose visual details by saving experience only as text. To address these limitations, we present a model-agnostic framework that allows frozen LLMs and VLMs to learn from deployment experience through three forms of external expertise: a Skill that guides reasoning and tool use, a Knowledge Memory that stores reliable facts supported by earlier cases or trusted external evidence, and a Multimodal Knowledge Base that keeps visual examples and guides the model to relate each retrieved case to the current image. Instead of relying on a fixed validation set, a validation strategy keeps an update only if it helps on new cases without degrading performance on earlier ones. Across six benchmarks covering clinical diagnosis, clinical workflows, medical reasoning, and medical and non-medical visual reasoning, and with four open-weight and closed-source base models, our framework improves performance during online deployment by up to 34.2% over the base model on medical tasks, generalizes to unseen cases, transfers to other models without further optimization, and works in non-medical domains.

Problem

• frozen models
• deployment experience
• multimodal medical AI
• continuous learning
• model-agnostic

Innovation

• Model-Agnostic Learning
• Frozen Models
• Multimodal Knowledge Base
• Deployment Experience
• Dynamic Validation Strategy