Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction

2026-07-17

AI Summary

This work addresses the performance degradation of vision-language models in transductive inference under severe class imbalance by proposing a dynamic shrinkage method grounded in penalized likelihood estimation. The approach leverages a von Mises–Fisher mixture model on the unit hypersphere to adaptively balance zero-shot priors and empirical estimates at both instance and class levels, thereby suppressing unreliable assignments and preventing aberrant class updates. The resulting framework is training-free, model-agnostic, and requires no task-specific hyperparameter tuning, yet consistently achieves substantial improvements in both accuracy and efficiency across diverse real-world imbalanced scenarios.

Abstract

A range of methods aim to enhance the performance of vision-language models (VLMs) at test time. Among them, transduction has emerged as a promising paradigm due to its strong compatibility and efficiency. However, realistic evaluations often involve highly imbalanced class distributions, which cause performance degradation or even collapse. In this work, we systematically revisit transduction from the perspective of penalized likelihood estimation (PLE), showing that PLE with a KL-divergence anchor term naturally yields an adaptive shrinkage behavior between prior anchors and empirical estimates. From this viewpoint, the brittleness of transductive methods can be attributed to the absence of anchoring mechanism and static modeling of the shrinkage strength. Therefore, we propose Mixture of Von Mises-Fisher Models with Dynamic Shrinkage (MOON). MOON is built upon a mixture of von Mises-Fisher distributions to model feature representations on the unit hypersphere. To handle imbalance, MOON dynamically adjusts the shrinkage strength using zero-shot priors at both instance and class levels. Thus, it suppresses unreliable assignments and prevents harmful updates from outlier classes, thereby mitigating negative transfer. MOON is model-agnostic, training-free, and requires no task-specific hyperparameter tuning. Extensive experiments further validate the advantage of MOON in both performance and efficiency. Our code is available at https://github.com/walawalagoose/MOON

Problem

• test-time transduction
• class imbalance
• vision-language models
• performance degradation
• negative transfer

Innovation

• von Mises-Fisher mixture
• dynamic shrinkage
• test-time transduction
• penalized likelihood estimation
• zero-shot priors