embedding design

This work challenges the conventional assumption that final-layer representations in large language models (LLMs) are optimal, revealing instead that intermediate-layer hidden states encode richer and more robust semantic information. Method: We propose the first multidimensional representation quality evaluation framework integrating information-theoretic measures (mutual information, compression ratio), manifold geometry, and perturbation invariance—designed for cross-architectural (Transformer/SSM) and cross-modal (text/vision) validation. Contribution/Results: Evaluated on 32 text embedding benchmarks, intermediate-layer embeddings consistently outperform final-layer counterparts by an average of 4.2%, demonstrating both statistical consistency and strong generalization across tasks and architectures. This study provides the first empirical evidence establishing the superiority of intermediate-layer representations, thereby introducing a new paradigm for efficient representation extraction, model compression, and interpretability research.

This work addresses the challenge of detecting generative AI–produced content. We propose an unsupervised, interpretable embedding-space analysis method: semantic embeddings of text or images are extracted using pre-trained large language or multimodal models; subsequently, dimensionality reduction (e.g., PCA) uncovers an intrinsic, low-dimensional distributional shift between AI-generated and human-created samples—rendering them highly separable without supervision. This phenomenon is systematically validated for the first time and endowed with human-interpretable semantic meaning (e.g., topic coherence, syntactic redundancy). Experiments across diverse generative models—including ChatGPT, Gemini, and Stable Diffusion—demonstrate that high-accuracy separation is achieved solely from raw embeddings and unsupervised projection, without fine-tuning, labeled data, or model-specific detectors. Our approach thus significantly enhances both generalizability and interpretability of AI-content detection.

Embedding models trained independently on similar data capture stable semantic meanings but yield inconsistent representation spaces, hindering interoperability across models. This work addresses compatibility challenges in multimodal search and model upgrades via orthogonal transformation-based embedding alignment. Theoretically, we derive the first tight Procrustes alignment error bound, proving the existence of a near-isometric orthogonal transformation that approximately preserves pairwise inner products—establishing rigorous theoretical foundations for alignment. Methodologically, we employ efficient Procrustes analysis as a post-hoc alignment procedure, preserving the intrinsic geometric structure of each embedding space while enabling cross-model alignment. Experiments demonstrate substantial improvements in model retraining compatibility, text retrieval fusion accuracy, and cross-modal search performance; our method achieves state-of-the-art results in hybrid multimodal search.

To address the challenge of predicting analytical operator outcomes on unseen, massive heterogeneous datasets, this paper introduces NumTabData2Vec—the first end-to-end dataset-level vectorization model. It maps raw tabular data to low-dimensional semantic embeddings, enabling cross-dataset semantic similarity search and analytical result inference. By jointly modeling data structure, statistical features, and operator semantics, NumTabData2Vec achieves high-accuracy outcome prediction on previously unseen, real-world multi-source datasets. Evaluated on multiple real-world benchmarks, it significantly outperforms state-of-the-art methods in prediction accuracy, reduces execution latency by over an order of magnitude, and effectively discriminates among diverse practical scenarios. This work establishes a scalable meta-learning paradigm for large-scale data analysis, offering robust generalization across heterogeneous datasets without requiring task-specific fine-tuning.

The evolution of embedding techniques from word vectors to multimodal representations remains fragmented, lacking a unified framework that integrates advances across linguistic, cross-lingual, personalized, and multimodal domains—particularly for embodied multimodal learning in large language models. Method: We systematically survey static and contextual language representations, cross-lingual and personalized modeling, sentence/document embeddings, and multimodal fusion in vision, robotics, and cognitive science. We synthesize recent progress in interpretability, model compression, numerical encoding, and bias mitigation, and propose a novel paradigm emphasizing strong alignment across non-textual modalities and scalable training. Contributions: We construct a comprehensive knowledge graph of end-to-end embedding technologies—from Word2Vec and BERT to GPT, generative topic models, and multimodal alignment/distillation methods—identifying key technical bottlenecks and ethical challenges. This work delivers the first systematic roadmap for multimodal, embodied learning in foundation models.

This study systematically investigates the intrinsic semantic and syntactic properties of mainstream word embedding methods—such as Word2Vec and GloVe—and their performance disparities across diverse natural language processing tasks. By establishing a unified evaluation framework that integrates publicly available pretrained embeddings with standard benchmark datasets, the work conducts empirical comparisons on canonical tasks including semantic similarity and analogical reasoning. The findings delineate the performance boundaries and optimal application scenarios for each embedding model, offering practitioners reliable guidance for model selection in real-world settings. Furthermore, the analysis deepens the understanding of the inherent limitations of static word representations, highlighting critical constraints in capturing contextual and compositional linguistic phenomena.

This work addresses the lack of a unified theoretical framework for assessing the reliability of nonlinear dimensionality reduction embeddings. It proposes a cohesive perspective grounded in differential and integral geometry, systematically analyzing the geometric properties of differentiable embeddings through both local differential structure and global path integrals. The study reveals, for the first time, that multiple existing diagnostic methods fundamentally arise from a common geometric object and demonstrates that their global characteristics cannot be fully captured by derivatives of any finite order, necessitating an irreducible integral viewpoint. Leveraging tools such as curvature analysis, path-dependence detection, and mapping continuity evaluation, the framework validates its theoretical predictions on both synthetic and real-world datasets—including single-cell data—enabling precise estimation of embedding reliability and effective discrimination between single-valued and path-dependent embeddings.

This study investigates whether latent representations from heterogeneous text embedding models can be transferred via simple transformations to enable direct AI-to-AI communication without decoding into human-readable text. For the first time, we systematically evaluate the effectiveness and limitations of linear mappings as lightweight translators across nine diverse models varying in architecture, pooling strategy, and training objective, using real-world textual data. Through comprehensive metrics—including Centered Kernel Alignment (CKA) similarity, downstream task transfer performance, fidelity, and retrieval accuracy—we find that simple transformations succeed only between partially compatible model pairs and largely fail otherwise. These results indicate that semantic transfer across heterogeneous embedding spaces cannot be universally achieved through alignment alone, as compatibility is jointly constrained by architectural design, training objectives, pooling mechanisms, and data distribution.