PETR: Prompt Ensembling with Training-free Routing for Vision-Language Models

2026-09-20

AI Summary

为解决视觉-语言模型在未见类别上泛化能力不足的问题，提出了一种基于无训练路由的提示集成方法(PETR)，通过双提示架构和统计相似性进行自适应路由。

Abstract

Prompt learning efficiently adapts vision-language models (VLMs) to downstream tasks, but gains on seen classes often come at the expense of generalization to unseen classes. To address this limitation, we propose prompt ensembling with training-free routing (PETR), whose key innovation is a carefully designed dual-prompt architecture: two complementary prompts are learned from different data and objectives to emphasize seen class discrimination and unseen-class generalization, respectively. During training, both prompts are fine-tuned using a shared frozen CLIP backbone, and statistical information is collected from the training set logits. At inference time, we determine the similarity of each test sample to seen data, and route the sample to the most appropriate prompt branch. To the best of our knowledge, this is the first prompt tuning framework that performs training-free adaptive routing based on statistical similarity. This design provides an interpretable routing signal and avoids common MoE-style routing pathologies, such as router training instability and load imbalance. Extensive experiments on 11 benchmark datasets demonstrate that our framework consistently outperforms previous methods on both seen and unseen classes, achieving new state-of-the-art results.

Problem

• Prompt Learning
• Vision-Language Models
• Generalization
• Seen Classes
• Unseen Classes

Innovation

• prompt ensembling
• training-free routing
• dual-prompt architecture
• statistical similarity