PERL: Parameter Efficient Reasoning in CLIP Latent Space

Unknown Date

AI Summary

This work addresses the challenge of efficiently adapting CLIP to downstream tasks while preserving its open-vocabulary generalization capability. The authors propose a lightweight adaptation framework that, for the first time, introduces an iterative implicit reasoning mechanism into vision-language models. By freezing the CLIP backbone and repeatedly injecting implicit reasoning tokens—generated by a compact, shared module—into intermediate layers, the method progressively refines semantic representations. Requiring only approximately 6K trainable parameters (up to 817× fewer than existing approaches), it achieves state-of-the-art parameter-efficiency trade-offs across 15 benchmarks and substantially improves few-shot accuracy on novel classes as well as transfer performance.

Abstract

Contrastively trained vision-language models such as CLIP provide strong zero-shot transfer by aligning images and text in a shared embedding space. However, adapting these models to downstream tasks without degrading their open-vocabulary generalization remains challenging. Existing parameter-efficient adaptation methods typically improve task specialization through learned prompts, adapters, or multimodal transformations, where adaptation capacity is primarily expressed through additional trainable parameters. Inspired by recent latent reasoning methods in language models, we investigate a complementary perspective: can adaptation emerge from iterative reasoning on latent representations rather than from increasing parameter count alone? We introduce PERL (Parameter-Efficient Reasoning in CLIP Latent Space), a lightweight adaptation framework that augments a frozen CLIP model with a compact shared reasoning module applied recurrently across refinement steps. At each step, PERL generates a latent reasoning token conditioned on the current representation and injects it into an intermediate encoder layer, progressively refining higher-level semantic representations while preserving CLIP's pretrained multimodal structure. Across 15 benchmarks spanning base-to-novel generalization, cross-dataset transfer, and out-of-distribution ImageNet variants, PERL achieves the best parameter-performance trade-off among the compared methods under a fast-adaptation few-shot setting, combining strong novel-class accuracy and competitive transfer performance with only about 6K trainable parameters, up to 817x fewer than the largest compared approach. Overall, our results suggest that iterative latent reasoning provides a complementary adaptation mechanism to parameter scaling in discriminative vision-language models.

Problem

• vision-language models
• parameter-efficient adaptation
• zero-shot transfer
• CLIP
• downstream task adaptation

Innovation

• parameter-efficient adaptation
• latent reasoning
• CLIP
• iterative refinement
• vision-language models