Automatic Detection of Inauthentic Templated Responses in English Language Assessments

2025-09-10

AI Summary

In high-stakes English proficiency testing, low-proficiency test-takers frequently resort to memorized essay templates to circumvent automated scoring systems, thereby compromising scoring fairness and validity. This paper formally introduces the task of Automated Detection of Template-Driven Responses (AuDITR), aimed at identifying inauthentic, highly formulaic responses. Methodologically, we design a set of multidimensional textual features—including syntactic repetitiveness, semantic rigidity, and lexical distribution anomalies—and integrate them into a lightweight machine learning classifier optimized for efficiency and adaptability to emerging template strategies. Experimental evaluation on authentic high-stakes test data yields an F1-score of 0.86, substantially outperforming established baselines. Our work contributes both a novel, well-defined detection task and a deployable technical framework to enhance the adversarial robustness of automated scoring systems against template-based cheating.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Feature Construction/Reformulation
• Data Mining & Knowledge Management: Anomaly/Outlier Detection

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

In high-stakes English Language Assessments, low-skill test takers may employ memorized materials called ``templates'' on essay questions to ``game'' or fool the automated scoring system. In this study, we introduce the automated detection of inauthentic, templated responses (AuDITR) task, describe a machine learning-based approach to this task and illustrate the importance of regularly updating these models in production.

Problem

• Detecting inauthentic templated responses in English assessments
• Identifying memorized materials used to fool automated scoring
• Developing machine learning models for template detection

Innovation

• Machine learning-based approach for detection
• Automated identification of templated responses
• Regular model updates in production systems