Benchmarking the Detection of LLMs-Generated Modern Chinese Poetry

2025-09-01

AI Summary

The increasing prevalence of AI-generated modern Chinese poetry poses significant challenges for authenticity verification, threatening the integrity of the poetic ecosystem. Method: We introduce the first dedicated benchmark for detecting large language model (LLM)-generated modern Chinese poems. It comprises high-quality poems authored by professional poets and four representative LLMs, systematically covering stylistic diversity and linguistic complexity. We further conduct the first comprehensive evaluation of six state-of-the-art text detectors—including both feature-based and deep learning–based approaches—on this benchmark. Results: Experimental results reveal a substantial performance degradation of existing detectors, exposing critical deficiencies in robustness and cross-model generalization. This work not only identifies unique challenges inherent to detecting modern Chinese poetry generation—such as stylistic subtlety and syntactic flexibility—but also provides a reproducible, extensible benchmark dataset and standardized evaluation framework, establishing a foundational resource for future research in AI-generated poetry detection.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Generative Adversarial Networks (GANs) for Vision

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

The rapid development of advanced large language models (LLMs) has made AI-generated text indistinguishable from human-written text. Previous work on detecting AI-generated text has made effective progress, but has not involved modern Chinese poetry. Due to the distinctive characteristics of modern Chinese poetry, it is difficult to identify whether a poem originated from humans or AI. The proliferation of AI-generated modern Chinese poetry has significantly disrupted the poetry ecosystem. Based on the urgency of identifying AI-generated poetry in the real Chinese world, this paper proposes a novel benchmark for detecting LLMs-generated modern Chinese poetry. We first construct a high-quality dataset, which includes both 800 poems written by six professional poets and 41,600 poems generated by four mainstream LLMs. Subsequently, we conduct systematic performance assessments of six detectors on this dataset. Experimental results demonstrate that current detectors cannot be used as reliable tools to detect modern Chinese poems generated by LLMs. The most difficult poetic features to detect are intrinsic qualities, especially style. The detection results verify the effectiveness and necessity of our proposed benchmark. Our work lays a foundation for future detection of AI-generated poetry.

Problem

• Detecting AI-generated modern Chinese poetry due to indistinguishability
• Addressing lack of benchmarks for LLM-generated poetry detection
• Identifying poetic features like style that challenge current detectors

Innovation

• Constructed high-quality dataset with human and AI poems
• Systematically assessed six detectors on modern Chinese poetry
• Identified style as most difficult poetic feature to detect