ReverseEOL: Improving Training-free Text Embeddings via Text Reversal in Decoder-only LLMs

2026-06-04

AI Summary

This work addresses the forward-context bias inherent in decoder-only large language models due to their causal attention mechanism, which prevents early tokens from accessing subsequent contextual information. To mitigate this limitation without requiring any training or fine-tuning, the authors propose ReverseEOL: a method that generates reversed embeddings by processing inverted input text and fuses them with the original forward embeddings. By integrating complementary bidirectional contextual cues, ReverseEOL substantially enhances the representational capacity of frozen models. The approach achieves significant improvements over existing training-free baselines on standard semantic textual similarity (STS) and MTEB benchmarks, demonstrating broad applicability across diverse architectures and model scales.

Abstract

Recent advances in Large Language Models (LLMs) have opened new avenues for generating training-free text embeddings. However, the causal attention in decoder-only LLMs prevents earlier tokens from attending to future context, leading to biased contextualized representations. In this work, we propose Reverse prompting with Explicit One-word Limitation (ReverseEOL), a simple yet effective method for enhancing the representational capability of frozen LLMs. ReverseEOL augments the standard forward embedding with an additional reversed embedding derived from the reversed input text. Since reversing the input exposes each token to context inaccessible in the original order, the resulting reversed embedding effectively provides complementary information to the original one. As a result, combining the forward and reversed embeddings yields a richer final representation. Comprehensive experiments on STS and MTEB benchmarks demonstrate that ReverseEOL significantly improves the performance of existing training-free baselines across a broad range of LLMs with diverse architectures and scales. Extensive ablations and analyses further confirm the necessity of our reversal mechanism.

Problem

• text embeddings
• decoder-only LLMs
• causal attention
• contextualized representations
• training-free

Innovation

• training-free embeddings
• text reversal
• decoder-only LLMs
• contextual representation
• ReverseEOL