Beyond Correctness: Evaluating and Improving LLM Feedback in Statistical Education

2025-11-10

AI Summary

This study investigates quality deficiencies in written feedback generated by large language models (LLMs) for statistics education—specifically, inaccuracies, insufficient pedagogical depth, and weak metacognitive scaffolding. Employing an extended pedagogical evaluation framework, we systematically compare four optimization approaches—zero-shot prompting, few-shot prompting, supervised fine-tuning, and LoRA—using GPT-series models as baselines. Results indicate that while all methods maintain baseline accuracy in diagnosis and explanation, contextualized feedback and higher-order learning guidance remain substantially limited. Zero-shot prompting achieves the optimal trade-off among feedback quality, generalizability, and deployment efficiency; fine-tuning strategies yield no significant improvement. We propose a low-cost, theory-grounded, and easily deployable LLM feedback optimization paradigm for statistics instruction. This work provides empirical evidence and methodological insights for AI-augmented statistical pedagogy.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Natural Language Processing: (Large) Language ModelsSearch and Optimization: Learning to Search

Application Category

User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendationSemantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Large language models (LLMs) have been proposed as scalable tools to address the gap between the importance of individualized written feedback and the practical challenges of providing it at scale. However, concerns persist regarding the accuracy, depth, and pedagogical value of their feedback responses. The present study investigates the extent to which LLMs can generate feedback that aligns with educational theory and compares techniques to improve their performance. Using mock in-class exam data from two consecutive years of an introductory statistics course at LMU Munich, we evaluated GPT-generated feedback against an established but expanded pedagogical framework. Four enhancement methods were compared in a highly standardized setting, making meaningful comparisons possible: Using a state-of-the-art model, zero-shot prompting, few-shot prompting, and supervised fine-tuning using Low-Rank Adaptation (LoRA). Results show that while all LLM setups reliably provided correctness judgments and explanations, their ability to deliver contextual feedback and suggestions on how students can monitor and regulate their own learning remained limited. Among the tested methods, zero-shot prompting achieved the strongest balance between quality and cost, while fine-tuning required substantially more resources without yielding clear advantages. For educators, this suggests that carefully designed prompts can substantially improve the usefulness of LLM feedback, making it a promising tool, particularly in large introductory courses where students would otherwise receive little or no written feedback.

Problem

• Evaluating LLM feedback accuracy and pedagogical value in statistics education
• Comparing enhancement methods for improving LLM feedback quality
• Addressing limitations in contextual feedback and student learning guidance

Innovation

• Evaluating GPT feedback with pedagogical framework
• Comparing four enhancement methods for LLMs
• Zero-shot prompting balances quality and cost