TRATES: Trait-Specific Rubric-Assisted Cross-Prompt Essay Scoring

2025-05-20

AI Summary

Existing automated essay scoring (AES) research predominantly focuses on holistic scores, neglecting fine-grained, cross-topic assessment of writing traits (e.g., logical coherence, lexical richness). Method: This paper introduces the first rubric-based, trait-specific AES framework, uniquely integrating LLM-driven, trait-oriented evaluation question generation with classical regression modeling. Leveraging prompt engineering, the LLM generates trait-specific assessment questions to extract transferable, trait-level features; a regression module then predicts dimension-wise scores. Contribution/Results: Our method achieves state-of-the-art performance across all trait dimensions on mainstream benchmarks. LLM-generated trait features exhibit the highest contribution to scoring accuracy, significantly enhancing cross-topic generalization, robustness against topic shift, and model interpretability—addressing critical limitations of prior holistic and trait-agnostic approaches.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Search and Optimization: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Research on holistic Automated Essay Scoring (AES) is long-dated; yet, there is a notable lack of attention for assessing essays according to individual traits. In this work, we propose TRATES, a novel trait-specific and rubric-based cross-prompt AES framework that is generic yet specific to the underlying trait. The framework leverages a Large Language Model (LLM) that utilizes the trait grading rubrics to generate trait-specific features (represented by assessment questions), then assesses those features given an essay. The trait-specific features are eventually combined with generic writing-quality and prompt-specific features to train a simple classical regression model that predicts trait scores of essays from an unseen prompt. Experiments show that TRATES achieves a new state-of-the-art performance across all traits on a widely-used dataset, with the generated LLM-based features being the most significant.

Problem

• Assessing essays by individual traits not holistic scores
• Leveraging LLM for trait-specific rubric-based essay scoring
• Combining trait-specific and generic features for cross-prompt prediction

Innovation

• Trait-specific rubric-assisted essay scoring framework
• LLM generates trait-specific assessment questions
• Combines features for cross-prompt regression model