SMCLM: Semantically Meaningful Causal Language Modeling for Autoregressive Paraphrase Generation

2025-07-04

AI Summary

This work addresses two key challenges in unsupervised paraphrase generation: low semantic fidelity and the limited reliability of automatic evaluation metrics (e.g., BLEU, ROUGE, BERTScore). To this end, we propose Semantic Causal Language Modeling (SCLM), a novel framework that integrates learnable semantic representations as initial embeddings into an autoregressive causal language modeling architecture—without requiring any parallel paraphrase data or supervised signals. SCLM employs self-supervised pretraining coupled with explicit semantic consistency constraints to jointly enhance semantic equivalence and lexical/structural diversity. Empirically, our method achieves state-of-the-art performance across multiple standard benchmarks for unsupervised paraphrasing. Furthermore, we conduct a rigorous diagnostic analysis revealing systematic biases in widely adopted automatic metrics, particularly their insensitivity to semantic equivalence under lexical variation. These findings provide critical insights for developing more trustworthy evaluation protocols in paraphrase generation research.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Causal Learning
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

This article introduces semantically meaningful causal language modeling (SMCLM), a selfsupervised method of training autoregressive models to generate semantically equivalent text. Our approach involves using semantically meaningful text representation as an initial embedding in the autoregressive training and generation processes. The extensive empirical study demonstrates that the SMCLM approach makes autoregressive models capable of learning robust and high-quality paraphrase generation. The proposed method is competitive with the supervised method and achieves state-of-the-art results in unsupervised approaches. This article also presents a comprehensive set of automatic metrics that cover a wide range of autogenerated paraphrase evaluation aspects. Simultaneously, this article highlights the low reliability of the metrics that are widely used in paraphrase generation evaluation, including BLEU, ROUGE, and BERTScore.

Problem

• Develops SMCLM for autoregressive paraphrase generation
• Improves semantic equivalence in unsupervised text generation
• Evaluates reliability of common paraphrase metrics critically

Innovation

• Self-supervised SMCLM for paraphrase generation
• Semantically meaningful initial embeddings usage
• State-of-the-art unsupervised paraphrase generation results