🤖 AI Summary
Vision-language models (VLMs) exhibit limited performance on medical imaging benchmarks and rely heavily on manual prompt engineering, hindering scalability and clinical deployment.
Method: We propose the first structured, automated prompt optimization framework tailored for medical VLMs—introducing DSPy to medical vision-language systems for the first time. Our framework enables end-to-end prompt automation without modifying model weights, abstracting prompt design into a modular pipeline and integrating four complementary optimization techniques. It supports five major medical imaging domains (radiology, gastroenterology, dermatology, pathology, and ophthalmology) and evaluates ten open-source VLMs.
Contribution/Results: Experiments show a median relative performance gain of 53% over zero-shot baselines, with task-specific improvements up to 300–3400%. The framework supports privacy-preserving local deployment and is fully open-sourced.
📝 Abstract
Vision-language foundation models (VLMs) show promise for diverse imaging tasks but often underperform on medical benchmarks. Prior efforts to improve performance include model finetuning, which requires large domain-specific datasets and significant compute, or manual prompt engineering, which is hard to generalize and often inaccessible to medical institutions seeking to deploy these tools. These challenges motivate interest in approaches that draw on a model's embedded knowledge while abstracting away dependence on human-designed prompts to enable scalable, weight-agnostic performance improvements. To explore this, we adapt the Declarative Self-improving Python (DSPy) framework for structured automated prompt optimization in medical vision-language systems through a comprehensive, formal evaluation. We implement prompting pipelines for five medical imaging tasks across radiology, gastroenterology, and dermatology, evaluating 10 open-source VLMs with four prompt optimization techniques. Optimized pipelines achieved a median relative improvement of 53% over zero-shot prompting baselines, with the largest gains ranging from 300% to 3,400% on tasks where zero-shot performance is low. These results highlight the substantial potential of applying automated prompt optimization to medical AI systems, demonstrating significant gains for vision-based applications requiring accurate clinical image interpretation. By reducing dependence on prompt design to elicit intended outputs, these techniques allow clinicians to focus on patient care and clinical decision-making. Furthermore, our experiments offer scalability and preserve data privacy, demonstrating performance improvement on open-source VLMs. We publicly release our evaluation pipelines to support reproducible research on specialized medical tasks, available at https://github.com/DaneshjouLab/prompt-triage-lab.