PRiSM: Prototype Regularization for Few-Shot VLMs

2026-07-20

AI Summary

Current training-free few-shot vision-language models suffer significant performance degradation under class-imbalanced and multi-class scenarios. To address this limitation, this work proposes PRiSM—a plug-and-play class prototype regularization module that enhances few-shot adaptability through a multi-objective loss function jointly optimizing inter-class distance maximization, support feature alignment, and prototype fidelity. The method innovatively employs a block-coordinate Majorize-Minimize optimizer grounded in the Gershgorin circle theorem for efficient optimization and introduces a more realistic evaluation benchmark based on Dirichlet sampling to better reflect practical data imbalances. Experimental results demonstrate that PRiSM consistently and substantially improves the performance of diverse training-free baseline methods, particularly in settings with a large number of classes and severe class imbalance.

Abstract

Training-free few-shot adaptation methods have gained significant attention recently in the context of Vision-language Models (VLMs). Yet, current benchmarks rely on strong assumptions about the statistics of the adaptation data, e.g., class balance. We question these simplifying assumptions and introduce a more realistic benchmark that varies both the levels of class balance and the effective number of classes in few-shot tasks via Dirichlet sampling. Surprisingly, under our setting, we observe substantial drops in the performances of state-of-the-art methods, more so when the number of labeled samples increases. To mitigate this, we introduce PRiSM, a class-prototype regularization that can be deployed as a plug and play module on top of any existing baseline method, significantly improving performances. Our method optimizes a novel multi-term loss, which includes a regularizer maximizing inter-class pairwise distances, along with additional terms promoting support-feature alignment and fidelity to the baseline prototypes. Furthermore, we introduce an effective and computationally efficient block Majorize-Minimize optimizer for our objective. More specifically, we derive a valid blockwise Lipschitz constant (i.e., a bound on the Hessian's spectral norm), which can be computed efficiently via the Gershgorin circle theorem. Extensive experiments show that PRiSM improves several training-free baselines, with large gains when dealing with severe class imbalance and high numbers of classes.

Problem

• few-shot learning
• vision-language models
• class imbalance
• prototype regularization
• training-free adaptation

Innovation

• prototype regularization
• few-shot learning
• vision-language models
• class imbalance
• Majorize-Minimize optimization