Contrastive Prompting Enhances Sentence Embeddings in LLMs through Inference-Time Steering

2025-05-19

AI Summary

Traditional prompt-based sentence embedding methods for large language models (LLMs) suffer from semantic impurity—specifically, encoding sentence semantics into the final token often incorporates redundant information such as stop words, degrading embedding quality. Method: We propose Contrastive Prompting (CP), a lightweight, inference-time prompting technique that introduces auxiliary contrastive prompts to steer LLMs toward focusing on core semantic content while suppressing irrelevant components. CP requires no fine-tuning, additional training data, or architectural modifications, and integrates seamlessly into existing prompt-based embedding frameworks. Contribution/Results: To our knowledge, this is the first work to incorporate contrastive learning principles into prompt-driven sentence embedding generation. CP yields substantial improvements on semantic textual similarity (STS) benchmarks and downstream classification tasks across multiple LLMs, demonstrating both effectiveness and strong generalizability without increasing computational overhead.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Extracting sentence embeddings from large language models (LLMs) is a practical direction, as it requires neither additional data nor fine-tuning. Previous studies usually focus on prompt engineering to guide LLMs to encode the core semantic information of the sentence into the embedding of the last token. However, the last token in these methods still encodes an excess of non-essential information, such as stop words, limiting its encoding capacity. To this end, we propose a Contrastive Prompting (CP) method that introduces an extra auxiliary prompt to elicit better sentence embedding. By contrasting with the auxiliary prompt, CP can steer existing prompts to encode the core semantics of the sentence, rather than non-essential information. CP is a plug-and-play inference-time intervention method that can be combined with various prompt-based methods. Extensive experiments on Semantic Textual Similarity (STS) tasks and downstream classification tasks demonstrate that our method can improve the performance of existing prompt-based methods across different LLMs. Our code will be released at https://github.com/zifengcheng/CP.

Problem

• Improving sentence embeddings in LLMs without fine-tuning
• Reducing non-essential information in last-token embeddings
• Enhancing semantic encoding via contrastive prompting at inference

Innovation

• Contrastive Prompting enhances sentence embeddings
• Steers prompts to encode core semantics
• Plug-and-play inference-time intervention method