Does a Large Language Model Really Speak in Human-Like Language?

2025-01-02

AI Summary

This study investigates whether large language model (LLM)-generated text exhibits the intrinsic population-level structural properties characteristic of human language. Method: We propose a hypothesis-testing framework based on cross-dataset semantic mapping, jointly embedding human-authored texts, LLM-generated paraphrases, and their second-order rewrites into a unified graph-structured latent space. Leveraging community detection and graph representation learning, we quantify statistical disparities in community distribution across the three text types. Contribution/Results: Our work introduces the first paraphrasing-aware, cross-text alignment paradigm for structural consistency assessment under semantic comparability. Empirical analysis reveals that GPT-series models consistently produce texts whose community structures significantly deviate from human writing (p < 0.01), rejecting the “human-like language” structural consistency hypothesis. This indicates that current LLMs lack the capacity to model the deep socio-cognitive structural regularities inherent in natural language.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Other Foundations of Human Computation & AI

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large Language Models (LLMs) have recently emerged, attracting considerable attention due to their ability to generate highly natural, human-like text. This study compares the latent community structures of LLM-generated text and human-written text within a hypothesis testing procedure. Specifically, we analyze three text sets: original human-written texts ($mathcal{O}$), their LLM-paraphrased versions ($mathcal{G}$), and a twice-paraphrased set ($mathcal{S}$) derived from $mathcal{G}$. Our analysis addresses two key questions: (1) Is the difference in latent community structures between $mathcal{O}$ and $mathcal{G}$ the same as that between $mathcal{G}$ and $mathcal{S}$? (2) Does $mathcal{G}$ become more similar to $mathcal{O}$ as the LLM parameter controlling text variability is adjusted? The first question is based on the assumption that if LLM-generated text truly resembles human language, then the gap between the pair ($mathcal{O}$, $mathcal{G}$) should be similar to that between the pair ($mathcal{G}$, $mathcal{S}$), as both pairs consist of an original text and its paraphrase. The second question examines whether the degree of similarity between LLM-generated and human text varies with changes in the breadth of text generation. To address these questions, we propose a statistical hypothesis testing framework that leverages the fact that each text has corresponding parts across all datasets due to their paraphrasing relationship. This relationship enables the mapping of one dataset's relative position to another, allowing two datasets to be mapped to a third dataset. As a result, both mapped datasets can be quantified with respect to the space characterized by the third dataset, facilitating a direct comparison between them. Our results indicate that GPT-generated text remains distinct from human-authored text.

Problem

• Large Language Models
• Human Text Features
• Parameter Adjustment

Innovation

• Large Language Models
• Human Text Similarity
• Parameter Tuning