Your Prompt Should Do More: Effects of Retrieval Instructions in Embedding Models

2026-10-07

AI Summary

This study addresses the challenge that existing embedding models struggle to reliably follow detailed instructions in asymmetric retrieval and remain susceptible to distractors on the query side. By analyzing the mechanisms through which instructions shape retrieval representations, it identifies training and evaluation deficiencies as the root causes of instruction-following failures. To overcome these limitations, this work proposes an adversarial fine-tuning strategy that explicitly introduces distractors at the query end to enhance robustness, coupled with a prompt embedding technique to optimize asymmetric retrieval representations. The proposed approach significantly improves the model’s ability to adhere to both simple and complex retrieval instructions while boosting overall retrieval performance, without compromising its effectiveness on other downstream tasks.

Abstract

Prompted embedding models have recently received increasing attention, particularly for retrieval, where detailed retrieval instructions are provided as part of the retrieval prompt. Several new datasets and studies have examined this setting, showing that the current embedding models often struggle to follow such instructions reliably. In this paper, we study the mechanism of how instructions actually affect the representations of retrieval queries in asymmetric retrieval tasks. We show that models can fail to follow even simple task instructions when query-side distractors are included in the evaluation. We hypothesize that this behavior is driven by the training setup of current embedding models and their evaluation, and show that fine-tuning with added query-side distractors leads to substantial improvements, with minimal effect on other tasks.

Problem

• embedding models
• retrieval instructions
• instruction following
• asymmetric retrieval
• query-side distractors

Innovation

• Prompted embedding models
• Retrieval instructions
• Query-side distractors
• Asymmetric retrieval
• Fine-tuning