GenAI Content Detection Task 3: Cross-Domain Machine-Generated Text Detection Challenge

2025-01-15

AI Summary

This study addresses the cross-domain, cross-model detection of large language model (LLM)-generated text—specifically, robust identification under a setting where training and test domains fully overlap yet span multiple domains and diverse LLM sources. Building upon the RAID benchmark, we present the first systematic validation of high-accuracy detection feasibility under fixed, diversified domain–LLM combinations. We propose a unified framework integrating feature extraction, discriminative classification, and domain adaptation to jointly process multi-source generated texts. Our approach incorporates BERT/RoBERTa-based encoders, contrastive learning, and ensemble strategies. Among 23 submissions from 9 teams, several models achieved >99% accuracy with false positive rates ≤5%. These results surpass conventional single-domain or open-domain detection paradigms, demonstrating both the practical viability and scalability of cross-domain, cross-model detection for LLM-generated content.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Recently there have been many shared tasks targeting the detection of generated text from Large Language Models (LLMs). However, these shared tasks tend to focus either on cases where text is limited to one particular domain or cases where text can be from many domains, some of which may not be seen during test time. In this shared task, using the newly released RAID benchmark, we aim to answer whether or not models can detect generated text from a large, yet fixed, number of domains and LLMs, all of which are seen during training. Over the course of three months, our task was attempted by 9 teams with 23 detector submissions. We find that multiple participants were able to obtain accuracies of over 99% on machine-generated text from RAID while maintaining a 5% False Positive Rate -- suggesting that detectors are able to robustly detect text from many domains and models simultaneously. We discuss potential interpretations of this result and provide directions for future research.

Problem

• Large Language Models
• Text Recognition
• Unseen Text Types

Innovation

• RAID
• Text Identification
• Large Language Models