🤖 AI Summary
This work addresses the inconsistent performance of binary quantization in embedding spaces, which excels in contrastive learning embeddings but degrades sharply in others, and resolves the lack of a unified theoretical foundation between the “random rotation” and “axis-aligned” quantization strategies. The study identifies the heterogeneity of coordinate-wise variances as the key factor governing quantization efficacy and establishes, for the first time, an analytical framework under a Gaussian structural assumption. This framework yields a closed-form solution for rank fidelity, quantitatively linking the information content of magnitude bits to variance heterogeneity, and unifies the conditions under which the two seemingly opposing strategies are optimal. Theoretical predictions are validated across 13 datasets and 6 embedding types, providing the first principled design guidelines for binary quantization systems.
📝 Abstract
Binary quantization (BQ) compresses high-dimensional embeddings into one or two bits per coordinate, enabling nearest neighbor search at extreme speed. Yet a striking puzzle persists: BQ achieves competitive recall on contrastive embeddings but fails on others -- and two leading systems adopt diametrically opposite strategies (random rotation vs. preserving coordinate axes) without a common theory explaining when each is appropriate. We resolve this puzzle by connecting the Gaussian structure recently established for InfoNCE-trained representations to a complete analytical framework for BQ quality. The key insight is that coordinate heterogeneity -- the non-uniformity of per-coordinate variances -- governs the key aspects of BQ performance. We derive closed-form expressions for ranking fidelity, prove that the magnitude bit carries information proportional to heterogeneity, and show that random rotation destroys precisely the signal that one paradigm exploits while creating the isotropy that the other requires. A two-parameter scaling law predicts fidelity across models and dimensions. Experiments on 13 datasets and 6 embedding families validate all predictions and provide the first principled design guide for binary quantization systems.