SeCo-SBIR: Semantically Consistent Prompt Learning for Zero-Shot Sketch-Based Image Retrieval

2026-08-04

AI Summary

This work addresses the challenge of balancing cross-domain adaptation and zero-shot generalization in zero-shot sketch-based image retrieval (ZS-SBIR). To this end, the authors propose a semantic-consistent prompt learning framework that effectively adapts the CLIP model through a text-guided intermediate representation injection mechanism and a perturbation-based asymmetric consistency constraint. This approach preserves CLIP’s inherent generalization capability while significantly enhancing retrieval performance. The framework incorporates learnable prompt vectors, a cross-modal coupling function, and a lightweight adapter module, optimized jointly via a multi-objective loss combining triplet, NT-Xent, and classification losses. Extensive experiments demonstrate that the proposed method achieves state-of-the-art performance across all three standard ZS-SBIR benchmarks—within-category, generalized, and cross-dataset settings.

Abstract

Adapting CLIP for zero-shot sketch-based image retrieval (ZS-SBIR) via prompt learning faces a fundamental tension: the model must bridge the sketch-photo domain gap through task-specific adaptation, yet the added flexibility risks overfitting to seen training categories and eroding CLIP's zero-shot generalization. We present SeCo-SBIR, a semantically consistent prompt learning framework that resolves this tension from both sides. First, a text-guided multi-modal prompting strategy routes learnable prompt vectors through CLIP's text encoder and projects the resulting intermediate representations into the visual encoder at every layer via learnable coupling functions. Because the text encoder has already learned robust, abstract category-level semantics from large-scale language supervision, this mechanism injects transferable semantic knowledge directly into the visual pathway - adapting the model to the sketch-photo domain while inherently favoring generalization to unseen classes. Second, a perturbation-based consistency constraint addresses the residual overfitting risk from the learnable coupling functions by aligning the adapted model with a frozen CLIP reference branch using an asymmetric InfoNCE objective - augmented inputs feed the frozen branch while clean inputs feed the trainable branch - anchoring the learned representations to CLIP's generalizable feature space. Together with lightweight adapters and a multi-objective loss combining triplet, NT-Xent, and classification terms, SeCo-SBIR achieves state-of-the-art results on all three standard ZS-SBIR benchmarks across categorical, generalized, and across-dataset settings.

Problem

• zero-shot sketch-based image retrieval
• domain gap
• prompt learning
• overfitting
• generalization

Innovation

• prompt learning
• zero-shot sketch-based image retrieval
• semantic consistency
• CLIP adaptation
• domain gap