Domain Regeneration: How well do LLMs match syntactic properties of text domains?

2025-05-12

AI Summary

This work investigates the syntactic fidelity of large language models (LLMs) in text regeneration—specifically, their ability to preserve human-authored syntactic properties. Method: Focusing on Wikipedia and news corpora—the two canonical training domains—we systematically assess multi-level syntactic features—including sentence length, readability, dependency label distributions, parsing depth, and tree complexity—under semantic controllability constraints. We adopt a corpus-linguistic observational paradigm, employing open-source LLMs with prompt-driven regeneration, spaCy/CoreNLP parsing, and multidimensional syntactic statistical analysis. Contribution/Results: We provide the first quantitative evidence of systematic syntactic distributional shifts in LLM-regenerated text. Results reveal pervasive mean shifts, variance compression, and long-tail attenuation across nearly all syntactic dimensions—indicating significant distributional degradation. These findings demonstrate that LLMs fundamentally fail to replicate the syntactic diversity characteristic of human language.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent improvement in large language model performance have, in all likelihood, been accompanied by improvement in how well they can approximate the distribution of their training data. In this work, we explore the following question: which properties of text domains do LLMs faithfully approximate, and how well do they do so? Applying observational approaches familiar from corpus linguistics, we prompt a commonly used, opensource LLM to regenerate text from two domains of permissively licensed English text which are often contained in LLM training data -- Wikipedia and news text. This regeneration paradigm allows us to investigate whether LLMs can faithfully match the original human text domains in a fairly semantically-controlled setting. We investigate varying levels of syntactic abstraction, from more simple properties like sentence length, and article readability, to more complex and higher order properties such as dependency tag distribution, parse depth, and parse complexity. We find that the majority of the regenerated distributions show a shifted mean, a lower standard deviation, and a reduction of the long tail, as compared to the human originals.

Problem

• How well LLMs match syntactic properties of text domains
• Faithfulness of LLMs in approximating text domain properties
• Comparison of regenerated and human text syntactic distributions

Innovation

• Regenerate text from Wikipedia and news domains
• Compare syntactic properties at varying abstraction levels
• Analyze shifted mean and reduced distribution tails