Exploring LLM-based Student Simulation for Metacognitive Cultivation

2025-02-17

AI Summary

Existing LLM-based student simulation approaches fail to authentically model learning difficulties and are constrained by the absence of validated assessment metrics and ethical limitations. This paper proposes a high-fidelity student simulation framework tailored for metacognitive education research: it constructs LLM-powered student agents that explicitly encode capability decay and domain-specific cognitive barriers. To enable rigorous, interpretable, and ethically compliant evaluation, we introduce a human-AI collaborative dual-round scoring mechanism and a graph-propagation-based grading algorithm. Experiments demonstrate that our method significantly improves the agent’s accuracy and generalizability in identifying authentic learning difficulties, isolates key cognitive determinants, and establishes a reliable simulation foundation for personalized pedagogical interventions and intelligent educational assessment.

Technology Category

• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Search and Optimization: Sampling/Simulation-based Search
• Multiagent Systems: Adversarial Agents

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Metacognitive education plays a crucial role in cultivating students' self-regulation and reflective thinking, providing essential support for those with learning difficulties through academic advising. Simulating students with insufficient learning capabilities using large language models offers a promising approach to refining pedagogical methods without ethical concerns. However, existing simulations often fail to authentically represent students' learning struggles and face challenges in evaluation due to the lack of reliable metrics and ethical constraints in data collection. To address these issues, we propose a pipeline for automatically generating and filtering high-quality simulated student agents. Our approach leverages a two-round automated scoring system validated by human experts and employs a score propagation module to obtain more consistent scores across the student graph. Experimental results demonstrate that our pipeline efficiently identifies high-quality student agents, and we discuss the traits that influence the simulation's effectiveness. By simulating students with varying degrees of learning difficulties, our work paves the way for broader applications in personalized learning and educational assessment.

Problem

• Simulating students with learning difficulties using LLMs
• Addressing lack of authentic student learning struggle representation
• Improving evaluation metrics for educational simulations

Innovation

• LLM-based student simulation
• two-round automated scoring
• score propagation module