TSM-Bench: Detecting LLM-Generated Text in Real-World Wikipedia Editing Practices

2026-05-29

AI Summary

Current benchmarks for detecting large language model–generated text struggle to identify task-specific machine-generated content commonly found in Wikipedia edits, leading to unreliable detection performance on real-world user-generated platforms. To address this gap, this work proposes TSM-Bench—the first multilingual, multi-generator, multi-task detection benchmark tailored to authentic editing scenarios, covering typical Wikipedia editing tasks such as summarization and paraphrasing. Systematic evaluation of state-of-the-art detectors reveals that models trained on generic data overfit to superficial artifacts, suffering accuracy drops of 10–40% on TSM-Bench. In contrast, task-specific fine-tuning generalizes effectively to generic data, but not vice versa, uncovering an asymmetry in generalization capability. This benchmark provides a more realistic foundation for developing robust detection models.

Abstract

Automatically detecting machine-generated text (MGT) is critical to maintaining the knowledge integrity of user-generated content (UGC) platforms such as Wikipedia. Existing detection benchmarks primarily focus on \textit{generic} text generation tasks (e.g., ``Write an article about machine learning.''). However, editors frequently employ LLMs for specific writing tasks (e.g., summarisation). These \textit{task-specific} MGT instances tend to resemble human-written text more closely due to their constrained task formulation and contextual conditioning. In this work, we show that a range of SOTA MGT detectors struggle to identify task-specific MGT reflecting real-world editing on Wikipedia. We introduce \textsc{TSM-Bench}, a multilingual, multi-generator, and \textit{multi-task} benchmark for evaluating MGT detectors on common, real-world Wikipedia editing tasks. Our findings demonstrate that (\textit{i}) average detection accuracy drops by 10--40\% compared to prior benchmarks, and (\textit{ii}) a generalisation asymmetry exists: fine-tuning on task-specific data enables generalisation to generic data -- even across domains -- but not vice versa. We demonstrate that models fine-tuned exclusively on generic MGT overfit to superficial artefacts of machine generation. Our results suggest that, in contrast to prior benchmarks, most detectors remain unreliable for automated detection in real-world contexts such as UGC platforms. \textsc{TSM-Bench} therefore provides a critical foundation for developing and evaluating future models.

Problem

• machine-generated text
• Wikipedia editing
• task-specific generation
• detection benchmark
• user-generated content

Innovation

• task-specific MGT detection
• TSM-Bench
• real-world editing
• generalisation asymmetry
• Wikipedia integrity