SKDU at De-Factify 4.0: Natural Language Features for AI-Generated Text-Detection

2025-03-28

AI Summary

To address the dual challenges of AI-text detection and generative-model attribution amid rapid LLM evolution, this paper proposes an end-to-end framework that jointly performs human/AI binary classification and fine-grained generator identification. The method integrates prompt-rewriting features inspired by RAIDAR with multi-dimensional content, stylistic, and propagation features extracted via the NELA toolkit, feeding them into an XGBoost classifier. Key contributions include: (1) the first systematic empirical validation that NELA features alone significantly outperform RAIDAR—even single NELA features surpass RAIDAR comprehensively—while their fusion yields only marginal gains, suggesting redundant low-discriminative signals; (2) establishing XGBoost as the optimal lightweight classifier for this task. Evaluated on the De-Factify 4.0 benchmark, the framework achieves 92.3% binary accuracy and 86.1% macro-F1 for multi-class generator attribution, demonstrating NELA’s strong capability in capturing fine-grained linguistic fingerprints.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Generative Adversarial Networks (GANs) for Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Large pretrained models with web data
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

The rapid advancement of large language models (LLMs) has introduced new challenges in distinguishing human-written text from AI-generated content. In this work, we explored a pipelined approach for AI-generated text detection that includes a feature extraction step (i.e. prompt-based rewriting features inspired by RAIDAR and content-based features derived from the NELA toolkit) followed by a classification module. Comprehensive experiments were conducted on the Defactify4.0 dataset, evaluating two tasks: binary classification to differentiate human-written and AI-generated text, and multi-class classification to identify the specific generative model used to generate the input text. Our findings reveal that NELA features significantly outperform RAIDAR features in both tasks, demonstrating their ability to capture nuanced linguistic, stylistic, and content-based differences. Combining RAIDAR and NELA features provided minimal improvement, highlighting the redundancy introduced by less discriminative features. Among the classifiers tested, XGBoost emerged as the most effective, leveraging the rich feature sets to achieve high accuracy and generalisation.

Problem

• Detecting AI-generated text using natural language features
• Comparing feature performance in binary and multi-class classification
• Identifying optimal classifiers for accurate AI-text detection

Innovation

• Pipelined approach with feature extraction
• NELA and RAIDAR features comparison
• XGBoost classifier for high accuracy