TNCSE: Tensor's Norm Constraints for Unsupervised Contrastive Learning of Sentence Embeddings

2025-03-17

AI Summary

Existing unsupervised sentence embedding methods model only directional similarity while neglecting magnitude information of embeddings, limiting representational capacity. This work introduces, for the first time, tensor norm constraints into unsupervised contrastive learning to explicitly enforce magnitude consistency between positive pairs, proposing the Magnitude-Aware Tensor-Norm Contrastive Sentence Embedding (TNCSE) framework—jointly optimizing both direction and magnitude. TNCSE integrates tensor norm regularization with a contrastive objective, requiring no labeled supervision. Evaluated on seven Semantic Textual Similarity (STS) benchmarks, it achieves state-of-the-art performance. In zero-shot transfer tasks, it consistently outperforms leading unsupervised baselines—including SimCSE and BERT-flow—demonstrating substantial improvements in generalization and discriminative power of unsupervised sentence representations.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Unsupervised & Self-Supervised Learning
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Unsupervised sentence embedding representation has become a hot research topic in natural language processing. As a tensor, sentence embedding has two critical properties: direction and norm. Existing works have been limited to constraining only the orientation of the samples' representations while ignoring the features of their module lengths. To address this issue, we propose a new training objective that optimizes the training of unsupervised contrastive learning by constraining the module length features between positive samples. We combine the training objective of Tensor's Norm Constraints with ensemble learning to propose a new Sentence Embedding representation framework, TNCSE. We evaluate seven semantic text similarity tasks, and the results show that TNCSE and derived models are the current state-of-the-art approach; in addition, we conduct extensive zero-shot evaluations, and the results show that TNCSE outperforms other baselines.

Problem

• Addresses unsupervised sentence embedding representation limitations
• Proposes Tensor's Norm Constraints for contrastive learning optimization
• Introduces TNCSE framework for state-of-the-art sentence embeddings

Innovation

• Tensor's Norm Constraints optimize unsupervised contrastive learning.
• Ensemble learning enhances Sentence Embedding representation framework.
• TNCSE achieves state-of-the-art in semantic text similarity tasks.