🤖 AI Summary
This study investigates how the geometric properties of text embedding spaces influence the choice of parameter-free similarity measures. Through a systematic evaluation of 19 similarity metrics across 19 encoders and 7 datasets, the work reveals— for the first time—that embedding anisotropy is a key determinant of optimal metric selection. The authors propose “single-dimension variance ratio” as a predictive indicator of metric performance: cosine similarity excels under isotropic conditions, whereas rank-based or L1-based metrics achieve approximately 20% higher performance under anisotropy. This indicator exhibits a strong linear correlation (r = 0.95) with metric advantage, and ablation via projection onto the principal direction eliminates this correlation, confirming its causal role.
📝 Abstract
The standard way to compare two text embeddings is cosine similarity. Scattered studies report that a different metric does better, but never pin down the geometric condition that decides when, or why. We settle both with a comprehensive empirical study: nineteen parameter-free similarity metrics on nineteen encoders, from compact sentence transformers up to seven-billion-parameter large language models, across seven datasets. The answer is geometric. When an encoder spreads its variance evenly across directions, cosine is the best parameter-free choice and no other metric helps by a usable margin. When the variance concentrates into a few dominant directions, a property known as anisotropy, rank-based and L1-type metrics beat cosine by a clear margin. The absolute gain is modest, but because cosine starts low on these encoders it is a sizable relative improvement, around twenty percent on average and largest where cosine is weakest. What decides this is the geometry of the embedding space, not how the model was trained: where the two disagree, the metric follows the geometry. One number, the fraction of variance held by the single most dominant dimension, predicts how much the alternatives help across all nineteen encoders, with a rank correlation of 0.86 and a linear correlation of 0.95. To test this as the cause rather than a correlate, we project out the dominant directions: cosine recovers and the advantage of the other metrics nearly vanishes, but only on the encoders that were anisotropic to begin with. The effect is directional, not magnitude based, since it survives normalizing every vector to unit length. Among parameter-free metrics, then, cosine is the right tool wherever an encoder is well spread, which includes the fine-tuned embedders commonly deployed for retrieval, and we give a one-number diagnostic for when it is not.