contrastive text-structure alignment

Current language models struggle to model the hierarchical nature of textual style, particularly exhibiting a fundamental limitation in synthesizing high-level stylistic attributes—such as formality or irony—from low-level features like lexical choice and punctuation. This work is the first to systematically challenge the implicit assumption that low-level style transfer suffices for high-level style representation. We argue that style embeddings must be semantically disentangled and propose a sentence-level style discrimination framework built upon BERT/RoBERTa, trained via contrastive learning. Experiments show that the learned embeddings effectively distinguish basic affective dimensions (e.g., valence, arousal) but exhibit significant generalization failure on higher-order styles—including formality and irony. Our findings expose structural deficiencies in current stylistic representation learning, both in underlying modeling assumptions and evaluation paradigms, offering both theoretical insights and an empirical benchmark for future research.

This work addresses the joint optimization of large language models (LLMs) and text embedding techniques to enhance efficiency and robustness in semantic matching, clustering, and information retrieval. We propose the first unified taxonomy centered on the *interaction patterns* between LLMs and embeddings—categorizing approaches into three paradigms: LLM-augmented embeddings, LLM-as-embedder, and LLM-understanding-embeddings—thereby transcending conventional task-centric taxonomies. By integrating supervised/unsupervised embedding learning, instruction tuning, prompt engineering, representation space analysis, and interpretability methods, we construct a structured knowledge graph encompassing over 100 studies. Our framework precisely delineates capability boundaries and application scopes for each paradigm, identifies persistent limitations inherited from pre-trained language models (PLMs) and novel challenges introduced by LLMs, and provides a theoretically grounded, empirically informed roadmap for future advancement.

This work investigates the fundamental trade-off between alignment—encompassing adversarial robustness and bias mitigation—and helpfulness—i.e., core task performance—in language models. We propose the first provable theoretical framework that quantifies how alignment and helpfulness vary with vector norm scaling in representation engineering: alignment improves linearly, whereas helpfulness degrades quadratically. Our analysis yields tight theoretical bounds on the alignment gain versus utility loss, explicitly characterizing the effective frontier and efficiency critical point of representation engineering. Systematic empirical evaluation across adversarial robustness, social bias reduction, and general capability benchmarks validates the theoretical predictions and delineates the feasible operational window for practical deployment. The core contribution lies in rigorously uncovering and formalizing this intrinsic trade-off, thereby providing both theoretical foundations and actionable guidance for controllable alignment in large language models.

The evolution of embedding techniques from word vectors to multimodal representations remains fragmented, lacking a unified framework that integrates advances across linguistic, cross-lingual, personalized, and multimodal domains—particularly for embodied multimodal learning in large language models. Method: We systematically survey static and contextual language representations, cross-lingual and personalized modeling, sentence/document embeddings, and multimodal fusion in vision, robotics, and cognitive science. We synthesize recent progress in interpretability, model compression, numerical encoding, and bias mitigation, and propose a novel paradigm emphasizing strong alignment across non-textual modalities and scalable training. Contributions: We construct a comprehensive knowledge graph of end-to-end embedding technologies—from Word2Vec and BERT to GPT, generative topic models, and multimodal alignment/distillation methods—identifying key technical bottlenecks and ethical challenges. This work delivers the first systematic roadmap for multimodal, embodied learning in foundation models.

Existing neural topic models face dual conflicts when jointly optimizing the evidence lower bound (ELBO) and contrastive learning objectives: ELBO prioritizes fine-grained reconstruction at the expense of semantic generalization, while document-level contrastive learning tends to capture low-level statistical noise (e.g., word frequency), hindering coherent topic discovery. To address this, we propose a set-level contrastive learning paradigm over topic vector collections—formulating neural topic modeling as a multi-objective optimization problem for the first time. We solve it via gradient-based Pareto-stationary optimization to jointly balance reconstruction fidelity and semantic generalization. Our method integrates a variational autoencoder, a set-level contrastive loss, and a topic-space alignment mechanism. Experiments on multiple benchmark datasets demonstrate consistent improvements: +3.2% in topic coherence, +5.1% in topic diversity, and up to +2.8% in downstream classification accuracy.

Current text-to-image generation methods face significant bottlenecks in semantic alignment accuracy and structural consistency. To address these limitations, we propose a dual-path optimization framework integrating contrastive alignment and structural guidance. First, a cross-modal contrastive learning module is introduced to enhance fine-grained semantic alignment within the CLIP embedding space. Second, structural priors—including layout maps and edge sketches—are explicitly incorporated to enforce geometric consistency. The framework jointly optimizes contrastive loss, structural reconstruction loss, and adversarial loss, improving generation controllability and fidelity without increasing inference overhead. Extensive experiments on COCO-2014 demonstrate state-of-the-art performance: +3.2% CLIP Score, −12.7% FID, and +5.8% SSIM over prior methods. Generated images exhibit superior semantic correctness and geometric integrity.

The formation mechanism and temporal evolution of lexical embedding geometry during language model training remain poorly understood. Method: We apply representational similarity analysis (RSA) to systematically track dynamic changes in the input embedding space of Pythia-12B and OLMo-7B, correlating embedding geometry with multidimensional linguistic metrics—including semantics, syntax, and word frequency—across training steps. Contribution/Results: We find that embedding geometry rapidly aligns with linguistic features within the first 1% of training steps. High-frequency function words converge significantly earlier than low-frequency content words, which retain stronger sensitivity to initialization randomness over extended training. This work provides the first empirical evidence that semantic–syntactic geometric organization of lexical embeddings emerges early and is shaped differentially by word frequency and part-of-speech functionality. Our findings establish a quantifiable geometric perspective on the origins of large language model capabilities, grounded in rigorous, time-resolved representational analysis.

Semantic matching of text requires jointly modeling hierarchical syntactic structure and fine-grained semantic distinctions, yet prevailing pretrained language models struggle to capture structured, cross-sentence interactions. To address this, we propose a context-aware dual-graph encoding framework: (1) constructing bilingual semantic graphs by integrating dependency parsing and topic modeling; (2) propagating structural features via Graph Isomorphism Networks (GIN); and (3) introducing a joint node-level and graph-level contrastive learning objective, enhanced by explicit and implicit negative sampling to refine the representation space. This work is the first to synergistically integrate structure-aware graph encoding with hierarchical contrastive learning for semantic matching. Evaluated on three legal document matching benchmarks and an academic plagiarism detection dataset, our method achieves state-of-the-art performance—e.g., 86.7% F1 on legal provision matching, representing an absolute improvement of 6.2%.

Current LLM embedding methods process only raw text, ignoring structural information such as hyperlinks and citations, resulting in insufficient contextual awareness. To address this, we propose a structure-aware endogenous encoding paradigm that intrinsically incorporates structural relations directly into the LLM’s encoding process—bypassing post-hoc aggregation. Methodologically, we design context distillation and semantic balancing mechanisms to suppress noise and systematically compare two structural integration strategies: sequential concatenation versus parallel caching. Leveraging zero-shot learning, our approach enables end-to-end structured modeling across retrieval, clustering, classification, and recommendation tasks. Experiments demonstrate consistent and significant improvements over both plain-text baselines and diverse post-processing approaches across multiple tasks, validating the method’s effectiveness, robustness, and scalability.

Concept alignment lacks a unified definition, and existing methods optimize divergent objectives under the same terminology, obscuring its fundamental nature. This work formalizes its multidimensional structure by decomposing it along two axes—“alignment targets” and “alignment levels”—and identifies four distinct alignment properties, revealing that current approaches satisfy only subsets of these. To address this limitation, we propose Coupled Sparse Autoencoders (CoSAE), a framework that jointly optimizes multiple alignment objectives, alongside InterVenchA, an interventional evaluation benchmark. Experiments demonstrate that optimizing a single objective fails to reliably recover other alignment properties, whereas CoSAE achieves strong instance-level conceptual consistency using merely 0.1% paired data.