entity embeddings

This work challenges the conventional assumption that final-layer representations in large language models (LLMs) are optimal, revealing instead that intermediate-layer hidden states encode richer and more robust semantic information. Method: We propose the first multidimensional representation quality evaluation framework integrating information-theoretic measures (mutual information, compression ratio), manifold geometry, and perturbation invariance—designed for cross-architectural (Transformer/SSM) and cross-modal (text/vision) validation. Contribution/Results: Evaluated on 32 text embedding benchmarks, intermediate-layer embeddings consistently outperform final-layer counterparts by an average of 4.2%, demonstrating both statistical consistency and strong generalization across tasks and architectures. This study provides the first empirical evidence establishing the superiority of intermediate-layer representations, thereby introducing a new paradigm for efficient representation extraction, model compression, and interpretability research.

In dense retrieval, single-layer document representations fail to fully capture the complementary linguistic knowledge distributed across layers of pretrained language models. To address this, we propose Multi-Layer Representation (MLR), a method that systematically leverages multi-layer encoder hidden states to construct more robust paragraph embeddings. MLR first analyzes layer-wise contributions to retrieval performance and then introduces a lightweight pooling strategy to compress multi-vector representations into efficient single vectors. Additionally, it integrates retrieval-oriented pretraining and hard negative mining to enhance discriminative capability. Experiments demonstrate that MLR significantly outperforms strong baselines—including Dual Encoder, ME-BERT, and ColBERT—under standard single-vector retrieval settings. It achieves state-of-the-art results on MS MARCO and Natural Questions, striking an effective balance between representational expressiveness and inference efficiency.

The evolution of embedding techniques from word vectors to multimodal representations remains fragmented, lacking a unified framework that integrates advances across linguistic, cross-lingual, personalized, and multimodal domains—particularly for embodied multimodal learning in large language models. Method: We systematically survey static and contextual language representations, cross-lingual and personalized modeling, sentence/document embeddings, and multimodal fusion in vision, robotics, and cognitive science. We synthesize recent progress in interpretability, model compression, numerical encoding, and bias mitigation, and propose a novel paradigm emphasizing strong alignment across non-textual modalities and scalable training. Contributions: We construct a comprehensive knowledge graph of end-to-end embedding technologies—from Word2Vec and BERT to GPT, generative topic models, and multimodal alignment/distillation methods—identifying key technical bottlenecks and ethical challenges. This work delivers the first systematic roadmap for multimodal, embodied learning in foundation models.

This work addresses the challenge of detecting generative AI–produced content. We propose an unsupervised, interpretable embedding-space analysis method: semantic embeddings of text or images are extracted using pre-trained large language or multimodal models; subsequently, dimensionality reduction (e.g., PCA) uncovers an intrinsic, low-dimensional distributional shift between AI-generated and human-created samples—rendering them highly separable without supervision. This phenomenon is systematically validated for the first time and endowed with human-interpretable semantic meaning (e.g., topic coherence, syntactic redundancy). Experiments across diverse generative models—including ChatGPT, Gemini, and Stable Diffusion—demonstrate that high-accuracy separation is achieved solely from raw embeddings and unsupervised projection, without fine-tuning, labeled data, or model-specific detectors. Our approach thus significantly enhances both generalizability and interpretability of AI-content detection.

This work investigates the learnability of high-dimensional embedding vectors from discrete data, focusing on how sample size, token frequency, and embedding–correlation strength jointly govern estimation accuracy. We propose a low-rank approximate Approximate Message Passing (AMP) algorithm grounded in a correlation–similarity coupled probabilistic model. This marks the first systematic integration of the AMP framework into the theoretical analysis of embedding estimation, enabling rigorous characterization of the phase transition boundary for estimation performance. Leveraging tools from high-dimensional statistical inference and random matrix theory, we derive precise quantitative relationships between embedding estimation error and key problem parameters. Extensive experiments on synthetic data and real-world text tasks validate our theoretical predictions, demonstrating substantial improvements in statistical efficiency and robustness—particularly in high-dimensional, sparse regimes.

This study systematically investigates the intrinsic semantic and syntactic properties of mainstream word embedding methods—such as Word2Vec and GloVe—and their performance disparities across diverse natural language processing tasks. By establishing a unified evaluation framework that integrates publicly available pretrained embeddings with standard benchmark datasets, the work conducts empirical comparisons on canonical tasks including semantic similarity and analogical reasoning. The findings delineate the performance boundaries and optimal application scenarios for each embedding model, offering practitioners reliable guidance for model selection in real-world settings. Furthermore, the analysis deepens the understanding of the inherent limitations of static word representations, highlighting critical constraints in capturing contextual and compositional linguistic phenomena.

This study investigates the impact of embedding dimensionality on performance in dense retrieval and its limitations as task complexity increases. Through systematic experiments across models of varying scales, the work presents the first empirical evidence that retrieval performance follows a power-law relationship with embedding dimensionality. Building on this observation, the authors propose predictable scaling laws based solely on dimensionality or jointly on model size. Using dense retrieval architectures, approximate nearest neighbor search, and large-scale comparative evaluations, they demonstrate that in task-aligned scenarios, performance improves with higher dimensionality—albeit with diminishing returns—whereas in misaligned tasks, excessive dimensions degrade performance. These findings offer both theoretical grounding and practical guidance for selecting optimal embedding dimensions in efficient retrieval systems.

Current large-scale text analysis relies either on costly large language models (LLMs) or dense embeddings lacking semantic controllability, hindering precise detection of model biases and data biases. This paper introduces an interpretable embedding method based on sparse autoencoders (SAEs), wherein embedding dimensions are explicitly aligned with human-understandable semantic concepts. The approach enables bias detection, concept association discovery, controllable clustering, and attribute-based retrieval. We present the first systematic evaluation demonstrating SAEs’ integrated advantages in interpretability, controllability, and cost-efficiency—enabling concept-level intervention, cross-model behavioral attribution, and training-data trigger pattern mining. Compared to LLM-based methods, our approach reduces computational cost by 2–8× while significantly improving bias identification reliability. It consistently outperforms dense embeddings across four benchmark tasks. Empirically, we localize Grok-4’s ambiguity-resolution tendency and identify training-data-triggered phrases in Tulu-3.

This work addresses the inherent tension between embedding expressiveness and serving efficiency in the candidate generation stage of recommender systems. The authors propose a novel training strategy that replaces conventional dense embeddings with high-dimensional sparse embeddings in the collaborative filtering autoencoder ELSA, enabling efficient learning of sparse representations in candidate retrieval models for the first time. The approach reduces embedding size by an order of magnitude without sacrificing accuracy—achieving only a 2.5% performance drop even under 100× compression—and simultaneously reveals an interpretable inverted index structure aligned with latent semantics. This structure naturally supports seamless integration of segment-level recommendations, such as those required for 2D homepage layouts, thereby jointly optimizing efficiency, representational capacity, and interpretability.

This work addresses a critical limitation in existing sentence embedding evaluation methods, which rely on downstream classifiers and thus conflate improvements in embedding quality with classifier-induced biases. To overcome this, the authors propose a classifier-free evaluation framework that quantifies how embeddings respond differently to syntactic noise and semantic negation injected into sentences. They introduce the novel “concept separation curve” to visualize a model’s ability to distinguish surface-level perturbations from genuine semantic changes. The approach is validated across multiple languages (English and Dutch), domains, and sentence lengths, demonstrating its effectiveness in providing an interpretable, reproducible, and model-agnostic assessment of conceptual stability in sentence embeddings. This significantly enhances the reliability and transparency of embedding quality evaluation.