Correcting Mean Bias in Text Embeddings: A Refined Renormalization with Training-Free Improvements on MMTEB

2025-11-14

AI Summary

Text embedding models exhibit systematic mean bias—outputs decompose into a sentence-invariant bias component and an effective semantic component. Method: We propose a plug-and-play, training-free renormalization technique grounded in theoretical analysis and empirical validation, demonstrating that vector projection subtraction outperforms direct mean subtraction for bias correction. The method requires only post-hoc processing of pretrained embeddings. Contribution/Results: Evaluated on 38 mainstream multilingual embedding models, it improves retrieval performance on the MTEB benchmark by 9.7σ, classification by 3.1σ, and other tasks by an average of 0.8σ. This work is the first to uncover the structural commonality of embedding bias across models and establishes an interpretable, general-purpose, training-free paradigm for enhancing embedding quality and semantic fidelity.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Computer Vision: Bias, Fairness & Privacy
• Machine Learning: Ethics, Bias, and Fairness

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We find that current text embedding models produce outputs with a consistent bias, i.e., each embedding vector $e$ can be decomposed as $ ilde{e} + mu$, where $mu$ is almost identical across all sentences. We propose a plug-and-play, training-free and lightweight solution called Renormalization. Through extensive experiments, we show that renormalization consistently and statistically significantly improves the performance of existing models on the Massive Multilingual Text Embedding Benchmark (MMTEB). In particular, across 38 models, renormalization improves performance by 9.7 $sigma$ on retrieval tasks, 3.1 $sigma$ on classification tasks, and 0.8 $sigma$ on other types of tasks. Renormalization has two variants: directly subtracting $mu$ from $e$, or subtracting the projection of $e$ onto $mu$. We theoretically predict that the latter performs better, and our experiments confirm this prediction.

Problem

• Correcting consistent mean bias in text embedding models
• Improving performance on multilingual embedding benchmarks
• Providing training-free bias removal through renormalization techniques

Innovation

• Renormalization corrects embedding bias without training
• Subtracts projection of vector onto bias direction
• Plug-and-play solution improves MMTEB benchmark performance