When to Unpair: Regulating Pairing Dependence in Medical Visual In-Context Learning

2026-10-07

AI Summary

This study addresses the issue of spurious correlations arising from excessive reliance on example pairing in medical visual in-context learning. To mitigate this, we propose a late unpairing curriculum learning mechanism that reveals the critical influence of training order on pairing dependency. By incorporating test-time label permutation and dynamic curriculum strategies to modulate the training sequence, our method effectively suppresses spurious features while preserving matching performance. Experimental results demonstrate that this mechanism significantly narrows the pairing gap, improving the Dice Similarity Coefficient (DSC) for BraTS segmentation from 0.733 to 0.857. Furthermore, it maintains cross-task generalization capabilities while alleviating bias, offering a novel paradigm for robust medical in-context learning.

Abstract

Visual in-context learning (ICL), well suited to label-scarce medical imaging, uses support image-label pairs to demonstrate input-output mappings, while the labels collectively indicate the requested task. We diagnose dependence on individual pairings with a test-time derangement that reassigns every support label to another support image while preserving the query, support images, and label multiset. The resulting pairing gap, defined as shuffled-minus-matched performance, shows that all four released models depend on the pairing, to widely varying degrees. Further analysis of a paired-trained model reveals support-associated spurious regions and lesion-size biases even with real, unaltered supports, alongside sensitivity to mis-registered support labels. To regulate this dependence, we introduce a late unpairing curriculum (LUC), which starts with matched training and then applies random unpairing, replacing each support label with that of another support in the same episode. LUC nearly closes the pairing gap on two backbones while maintaining or improving matched-support performance across all evaluated task types, with gains extending to held-out tasks and cross-dataset episodes. It also mitigates these failure modes. On BraTS whole-tumor segmentation, matched-support DSC rises from 0.733 to 0.857 while the gap shrinks from -0.184 to -0.008. In a released model, brief fine-tuning with random unpairing reduces the gap. A reversed curriculum that places the same number of unpairing epochs at the start of training leaves a large gap. This shows that pairing dependence is shaped by the order of training and not only by the amount of unpaired training.

Problem

• visual in-context learning
• pairing dependence
• medical imaging
• spurious correlations
• robustness

Innovation

• Visual In-Context Learning
• Late Unpairing Curriculum
• Pairing Dependence
• Test-time Derangement
• Medical Image Segmentation