GenAI Content Detection Task 2: AI vs. Human -- Academic Essay Authenticity Challenge

2024-12-24

AI Summary

This study addresses the growing challenge of distinguishing human-written from AI-generated academic text. We introduce the first bilingual (English–Arabic) benchmark for academic text authenticity evaluation and launch a cross-lingual detection challenge under the COLING 2025 framework. Methodologically, we propose a fine-tuned Transformer-based detection backbone enhanced with semantic representations from large language models (e.g., Llama 2/3), integrated within a multi-dimensional evaluation framework that benchmarks against n-gram baselines. Our key contributions are: (1) releasing a high-quality, domain-adapted bilingual annotated dataset; (2) establishing a standardized definition for cross-lingual AI-text detection in academic contexts; and (3) empirically validating the strong generalization capability of state-of-the-art models—our best-performing system achieves F1-scores exceeding 0.98 on both English and Arabic test sets, substantially outperforming conventional approaches.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: AI for Accessibility

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

This paper presents a comprehensive overview of the first edition of the Academic Essay Authenticity Challenge, organized as part of the GenAI Content Detection shared tasks collocated with COLING 2025. This challenge focuses on detecting machine-generated vs. human-authored essays for academic purposes. The task is defined as follows:"Given an essay, identify whether it is generated by a machine or authored by a human.'' The challenge involves two languages: English and Arabic. During the evaluation phase, 25 teams submitted systems for English and 21 teams for Arabic, reflecting substantial interest in the task. Finally, seven teams submitted system description papers. The majority of submissions utilized fine-tuned transformer-based models, with one team employing Large Language Models (LLMs) such as Llama 2 and Llama 3. This paper outlines the task formulation, details the dataset construction process, and explains the evaluation framework. Additionally, we present a summary of the approaches adopted by participating teams. Nearly all submitted systems outperformed the n-gram-based baseline, with the top-performing systems achieving F1 scores exceeding 0.98 for both languages, indicating significant progress in the detection of machine-generated text.

Problem

• AI-generated content
• human writing
• text classification

Innovation

• Academic Essay Authenticity Challenge
• Transformer-based Models
• Content Detection Technology