'Your AI Text is not Mine': Redefining and Evaluating AI-generated Text Detection under Realistic Assumptions

2026-06-03

AI Summary

This study addresses the lack of a unified definition of “harmful use” in current AI-generated text detection research and challenges the unrealistic assumptions commonly adopted. We systematically define multiple types of AI-generated text and propose the first fine-grained text provenance annotation framework tailored to real-world human-AI collaboration scenarios. To support this, we introduce AITDNA, a novel benchmark dataset that captures complete editing and interaction histories. Using AITDNA, we conduct a systematic evaluation of state-of-the-art detection models and reveal their limited generalization—performing reliably only on specific text types. Our work shifts the focus of detection research from idealized settings toward practical applications and publicly releases the AITDNA dataset and associated code.

Abstract

Although it is generally agreed that AI-generated text poses a broad societal risk, there is no common understanding in the AI-generated text detection literature on what constitutes harmful use. Rather, existing datasets and approaches often define their own criteria and make their own assumptions, sometimes implicitly, and often only loosely related to real-world needs and applications. To address this gap, we here systematically define various notions of AI-generated text and their characteristics. To study these, we collect AITDNA - a new benchmark of human-machine co-constructed texts that is annotated with detailed genesis information, such as the entire edit and AI-interaction history. We benchmark various machine-generated text detectors and find that they often only perform well for specific notions but not as broad detectors. We release code and data publicly.

Problem

• AI-generated text detection
• harmful use
• realistic assumptions
• benchmark dataset
• generalization

Innovation

• AI-generated text detection
• realistic assumptions
• human-AI co-authoring
• benchmark dataset
• text provenance