CAFES: A Collaborative Multi-Agent Framework for Multi-Granular Multimodal Essay Scoring

2025-05-20

AI Summary

Traditional automated essay scoring (AES) suffers from poor generalizability and incomplete multimodal perception, while existing multimodal large language model (MLLM)-based approaches exhibit low agreement with human judgments. To address these limitations, this paper proposes CAFES—the first collaborative multi-agent framework tailored for AES. CAFES implements a three-stage closed-loop process: initial scoring, feedback aggregation, and reflective refinement—enabling trait-specific, multi-granularity scoring, structured feedback pool management, and iterative self-reflection. Its key contributions are: (1) the first multi-agent collaborative architecture for AES; (2) the first evidence-driven, multimodal AES closed-loop optimization paradigm; and (3) significantly improved alignment with human scoring. Experiments demonstrate that CAFES achieves an average 21% improvement in quadratic weighted kappa (QWK) across multiple metrics, with particularly notable gains in grammatical accuracy and lexical diversity.

Technology Category

• Multiagent Systems: Teamwork
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Automated Essay Scoring (AES) is crucial for modern education, particularly with the increasing prevalence of multimodal assessments. However, traditional AES methods struggle with evaluation generalizability and multimodal perception, while even recent Multimodal Large Language Model (MLLM)-based approaches can produce hallucinated justifications and scores misaligned with human judgment. To address the limitations, we introduce CAFES, the first collaborative multi-agent framework specifically designed for AES. It orchestrates three specialized agents: an Initial Scorer for rapid, trait-specific evaluations; a Feedback Pool Manager to aggregate detailed, evidence-grounded strengths; and a Reflective Scorer that iteratively refines scores based on this feedback to enhance human alignment. Extensive experiments, using state-of-the-art MLLMs, achieve an average relative improvement of 21% in Quadratic Weighted Kappa (QWK) against ground truth, especially for grammatical and lexical diversity. Our proposed CAFES framework paves the way for an intelligent multimodal AES system. The code will be available upon acceptance.

Problem

• Improves generalizability in automated essay scoring
• Addresses misalignment between AI and human scoring
• Enhances multimodal perception for diverse essay traits

Innovation

• Collaborative multi-agent framework for AES
• Specialized agents for iterative scoring
• 21% QWK improvement with MLLMs