Technology Category
Natural Language Processing: GenerationMachine Learning: Large Multimodal Models (LMMs)Planning, Routing, and Scheduling: Planning with Language Models
On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey
🤖 AI Summary
Existing general-purpose text embedding (GPTE) methods exhibit fragmented understanding of pre-trained language models’ (PLMs) roles and rely on narrow optimization objectives. Method: We propose a multi-level framework categorizing PLMs’ roles across architecture design, representation enhancement, training paradigms, and data construction; extend the scope to emerging challenges—including safety, bias mitigation, and cognitive scalability—and integrate PLM-driven dense vector generation, contrastive learning, and large-scale pairwise supervision to support multilingual, multimodal, and code embedding. Contribution/Results: This work delivers the first structured technical survey and developmental roadmap for GPTE, systematically clarifying PLMs’ evolving functions. It significantly improves generalization and robustness across downstream tasks—including retrieval, classification, and clustering—while unifying disparate methodological advances under a coherent conceptual framework.
Technology Category
Application Category
📝 Abstract
Text embeddings have attracted growing interest due to their effectiveness across a wide range of natural language processing (NLP) tasks, such as retrieval, classification, clustering, bitext mining, and summarization. With the emergence of pretrained language models (PLMs), general-purpose text embeddings (GPTE) have gained significant traction for their ability to produce rich, transferable representations. The general architecture of GPTE typically leverages PLMs to derive dense text representations, which are then optimized through contrastive learning on large-scale pairwise datasets. In this survey, we provide a comprehensive overview of GPTE in the era of PLMs, focusing on the roles PLMs play in driving its development. We first examine the fundamental architecture and describe the basic roles of PLMs in GPTE, i.e., embedding extraction, expressivity enhancement, training strategies, learning objectives, and data construction. Then, we describe advanced roles enabled by PLMs, such as multilingual support, multimodal integration, code understanding, and scenario-specific adaptation. Finally, we highlight potential future research directions that move beyond traditional improvement goals, including ranking integration, safety considerations, bias mitigation, structural information incorporation, and the cognitive extension of embeddings. This survey aims to serve as a valuable reference for both newcomers and established researchers seeking to understand the current state and future potential of GPTE.
Problem
Research questions and friction points this paper is trying to address.
Explores PLMs' role in general-purpose text embeddings development
Analyzes PLMs' impact on multilingual and multimodal embedding capabilities
Identifies future research directions for text embedding enhancements
Innovation
Methods, ideas, or system contributions that make the work stand out.
Leverages pretrained language models for embeddings
Uses contrastive learning for optimization
Enhances multilingual and multimodal integration
🔎 Similar Papers
No similar papers found.
