Human-AI Collaboration or Academic Misconduct? Measuring AI Use in Student Writing Through Stylometric Evidence

2025-05-13

AI Summary

The growing prevalence of human-AI collaboration in education lacks interpretable, quantitative tools for assessing AI involvement in student writing—particularly tools that support pedagogical goals rather than merely detecting misconduct. Method: This paper proposes an authorship verification framework for academic writing that transparently and constructively quantifies the degree of AI assistance. It introduces a novel stylometric approach—Authorship Vector (AV) differential analysis—to enable fine-grained, word- and sentence-level quantification of AI intervention intensity and evaluates robustness against paraphrasing by mainstream LLMs. Contribution/Results: Validated on author identification tasks using stylometric analysis and multi-source real-world data—including PAN-14 and 1,889 student assignments from 540 undergraduates at the University of Melbourne—the framework achieves high accuracy and interpretability. It provides educators with an actionable, pedagogically grounded tool to foster responsible AI-assisted writing and enhance students’ metacognitive development.

Technology Category

• Natural Language Processing: Sentiment Analysis, Stylistic Analysis, and Argument Mining
• Humans and AI: Human-AI Collaboration / Human-AI Teaming
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

As human-AI collaboration becomes increasingly prevalent in educational contexts, understanding and measuring the extent and nature of such interactions pose significant challenges. This research investigates the use of authorship verification (AV) techniques not as a punitive measure, but as a means to quantify AI assistance in academic writing, with a focus on promoting transparency, interpretability, and student development. Building on prior work, we structured our investigation into three stages: dataset selection and expansion, AV method development, and systematic evaluation. Using three datasets - including a public dataset (PAN-14) and two from University of Melbourne students from various courses - we expanded the data to include LLM-generated texts, totalling 1,889 documents and 540 authorship problems from 506 students. We developed an adapted Feature Vector Difference AV methodology to construct robust academic writing profiles for students, designed to capture meaningful, individual characteristics of their writing. The method's effectiveness was evaluated across multiple scenarios, including distinguishing between student-authored and LLM-generated texts and testing resilience against LLMs' attempts to mimic student writing styles. Results demonstrate the enhanced AV classifier's ability to identify stylometric discrepancies and measure human-AI collaboration at word and sentence levels while providing educators with a transparent tool to support academic integrity investigations. This work advances AV technology, offering actionable insights into the dynamics of academic writing in an AI-driven era.

Problem

• Measure AI assistance in student writing using stylometric evidence
• Distinguish student-authored from AI-generated academic texts
• Promote transparency in human-AI collaboration for academic integrity

Innovation

• Adapted Feature Vector Difference AV methodology
• Stylometric analysis for AI-generated text detection
• Transparent tool for academic integrity investigations