🤖 AI Summary
In few-shot domain adaptation, two key challenges hinder performance: (1) hyperparameter tuning is impractical due to scarce validation data, and (2) models lack robustness under distribution shift. To address these, we propose Soup-Adapter—a hyperparameter-agnostic, distributionally robust adapter ensemble method. Our approach extends the CLIP adapter paradigm to DINOv2 for the first time and introduces a reparameterizable adapter soup: multiple adapters are trained independently via distinct paths, their outputs are averaged at inference, and their parameters are concatenated and reparameterized to stabilize optimization. Experiments demonstrate that Soup-Adapter consistently outperforms single-adapter baselines across a wide range of hyperparameters and significantly improves both accuracy and robustness on cross-domain few-shot tasks. This work establishes a new, efficient, practical, and generalizable paradigm for few-shot domain adaptation.
📝 Abstract
In this paper, we tackle two fundamental problems in few-shot domain adaptation of foundation models. First, hyperparameter tuning is often impractical due to the lack of large validation datasets. Second, model robustness under distribution shifts where test time data deviates slightly from training distributions, remains a concern. We show that by training multiple independent adapters and averaging their outputs, the new model has a higher performance and is more robust to distribution shifts compared to any individual adapter. This improvement holds even when the adapters are trained with diverse hyperparameters sampled from a wide range, resulting in varied individual performance. Consequently, our method addresses both of the problems described above. The ensemble is also significantly less sensitive to the residual ratio, a critical hyperparameter of CLIP-Adapter. Since the ensemble can be reparameterized to a single adapter again using a principled concatenation of the parameters, we refer to our method as Soup-Adapter. This is also the first study to explore CLIP adapter-style techniques for DINOv2 and to directly compare them with CLIP in this setting.