Prompt-Anchored Residual Adaptation for Biomedical Vision-Language Models

2026-09-27

AI Summary

This study addresses the prediction dependency and performance instability arising from support set composition during few-shot adaptation of biomedical vision-language models. To this end, we propose PARA, a method that retains frozen prompts as semantic anchors and introduces an anchored relative residual mechanism. By correcting support set visual predictions through closed-form residual computation, PARA achieves robust few-shot classification adaptation. Furthermore, we design a repeated support set evaluation protocol that effectively decouples selection variance from optimization stochasticity, thereby enhancing evaluation reliability. Experimental results demonstrate that PARA achieves state-of-the-art performance on both few-shot classification and base-to-new generalization tasks.

Abstract

Pretrained biomedical vision-language models achieve strong zero-shot performance in biomedical image classification. However, downstream biomedical classification often depends on subtle visual differences between classes that may not be fully captured by pretrained representations. Few-shot adaptation addresses this mismatch by optimizing a task-specific predictor on a small labeled support set. Because the selected examples capture only part of the visual variation within the target classes, the adapted predictions can depend strongly on their composition. We propose Prompt-Anchored Residual Adaptation (PARA), which retains the frozen prompt prediction as a support-invariant semantic anchor and incorporates a visual prediction learned from the support set through an anchor-relative residual. The residual step is computed in a closed form from frozen support embeddings using anchor discrepancy and support agreement. Support-set dependence also limits evaluation: comparisons are fair within a shared draw but remain conditional on its composition. To obtain more reliable comparisons, we introduce a repeated-support protocol that separates support-selection variation from optimization randomness and reports both average and worst-20% performance. PARA achieves state-of-the-art performance in both few-shot classification and base-to-novel generalization.

Problem

• biomedical vision-language models
• few-shot adaptation
• support-set dependence
• image classification
• evaluation protocol

Innovation

• Prompt-Anchored Residual Adaptation
• Few-shot adaptation
• Biomedical vision-language models
• Closed-form residual
• Repeated-support protocol