Map of Encoders -- Mapping Sentence Encoders using Quantum Relative Entropy

2026-02-09

AI Summary

This work proposes a novel encoder mapping framework to address the challenge of effectively comparing and visualizing relationships among large-scale sentence encoders. For the first time, quantum relative entropy (QRE) is introduced into sentence encoder analysis, integrated with sentence embedding matrices, pairwise inner product (PIP) matrices, and their eigenvectors to systematically map 1,101 publicly available encoders. The resulting map not only reveals the proximity structure of encoders in representation space but also accurately predicts their downstream performance on tasks such as retrieval and clustering. This approach provides a global, interpretable, and visualization-driven framework for understanding and selecting sentence encoders, offering both theoretical insight and practical utility for the NLP community.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Representation Learning
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

We propose a method to compare and visualise sentence encoders at scale by creating a map of encoders where each sentence encoder is represented in relation to the other sentence encoders. Specifically, we first represent a sentence encoder using an embedding matrix of a sentence set, where each row corresponds to the embedding of a sentence. Next, we compute the Pairwise Inner Product (PIP) matrix for a sentence encoder using its embedding matrix. Finally, we create a feature vector for each sentence encoder reflecting its Quantum Relative Entropy (QRE) with respect to a unit base encoder. We construct a map of encoders covering 1101 publicly available sentence encoders, providing a new perspective of the landscape of the pre-trained sentence encoders. Our map accurately reflects various relationships between encoders, where encoders with similar attributes are proximally located on the map. Moreover, our encoder feature vectors can be used to accurately infer downstream task performance of the encoders, such as in retrieval and clustering tasks, demonstrating the faithfulness of our map.

Problem

• sentence encoders
• quantum relative entropy
• embedding comparison
• encoder visualization
• pre-trained models

Innovation

• Quantum Relative Entropy
• Sentence Encoder Mapping
• Pairwise Inner Product
• Embedding Visualization
• Encoder Comparison