Operation-Guided Progressive Human-to-AI Text Transformation Benchmark for Multi-Granularity AI-Text Detection

2026-06-04

AI Summary

This work addresses a critical gap in existing AI-generated text detection benchmarks, which overlook the dynamic evolution of human-AI collaborative editing and fail to capture multi-granular authorship signals. To this end, we introduce OpAI-Bench, the first operation-guided, progressive human-AI mixed text benchmark. Starting from human-written source documents, it generates nine incremental versions across four domains through five representative AI editing operations, while preserving full provenance annotations at document, sentence, token, and span levels. Our experiments reveal that detectability of AI involvement depends not only on the proportion of AI-generated content but is also jointly influenced by editing operation type, domain, and revision trajectory. Notably, intermediate hybrid versions prove more challenging to detect than purely human or heavily AI-edited texts. OpAI-Bench thus provides a controlled, fine-grained platform for evaluating and understanding the detectability of AI-assisted writing.

Abstract

As AI writing assistants become increasingly integrated into real-world drafting and revision workflows, many documents are no longer purely human-written or AI-generated, but instead result from progressive human-AI co-editing. However, existing AI-text detection benchmarks largely focus on final outputs and provide limited understanding of how AI authorship signals emerge, accumulate, or disappear throughout the revision process. We introduce OpAI-Bench, an operation-guided benchmark for studying progressive human-to-AI text transformation across document, sentence, token, and span granularities. Starting from human-written documents, OpAI-Bench constructs nine sequentially revised versions for each sample under predefined AI coverage levels and five representative AI edit operations, covering four domains while preserving complete authorship provenance at multiple granularities. The benchmark supports comprehensive evaluation with 8 document-level detectors, 7 sentence-level detectors, and 2 fine-grained token/span-level detectors. Experiments reveal that AI-text detectability is governed not only by the proportion of AI-edited content, but also by edit operation, domain, and cumulative revision history. Interestingly, we notice that mixed-authorship intermediate versions are often harder to detect than both fully human and heavily AI-edited endpoints, exposing non-monotonic detection patterns missed by existing benchmarks. OpAI-Bench provides a controlled testbed for analyzing whether, when, and how AI-assisted writing becomes detectable under realistic progressive editing scenarios. Our code and benchmark are available at https://github.com/VILA-Lab/OpAI-Bench.

Problem

• AI-text detection
• human-AI co-editing
• progressive editing
• multi-granularity
• authorship provenance

Innovation

• progressive editing
• multi-granularity detection
• AI-text provenance
• operation-guided benchmark
• human-AI co-editing