Automated Formative Feedback for Short-form Writing: An LLM-Driven Approach and Adoption Analysis

2025-09-25

AI Summary

This study addresses three challenges in engineering capstone courses: inconsistent student biweekly presentation quality, high instructor workload for manual feedback, and low initial student acceptance of AI-generated feedback. To address these, we developed a formative feedback system powered by large language models (LLMs), integrating an educational task-parsing algorithm with a personalized feedback generation mechanism. The system automatically detects structural deficiencies, content incompleteness, and missing key deliverables in student reports, and provides actionable, improvement-oriented suggestions. Empirical evaluation shows that although initial adoption was low, sustained users demonstrated statistically significant improvements in report completeness and expressive quality (p < 0.01). Furthermore, the system enables instructors and program administrators to gain novel, process-oriented insights into student learning progression. This work validates the effectiveness and practical feasibility of lightweight, context-aware, LLM-driven feedback tools for advanced engineering writing instruction.

Technology Category

Natural Language Processing: Prompt Engineering / PromptingMachine Learning: Large Multimodal Models (LMMs)Humans and AI: Teamwork, Team formation

Application Category

User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendationEconomics, Online Markets and Human Computation: Cost models of using LLMs in production systemsSearch and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

This paper explores the development and adoption of AI-based formative feedback in the context of biweekly reports in an engineering Capstone program. Each student is required to write a short report detailing their individual accomplishments over the past two weeks, which is then assessed by their advising professor. An LLM-powered tool was developed to provide students with personalized feedback on their draft reports, guiding them toward improved completeness and quality. Usage data across two rounds revealed an initial barrier to adoption, with low engagement rates. However, students who engaged in the AI feedback system demonstrated the ability to use it effectively, leading to improvements in the completeness and quality of their reports. Furthermore, the tool's task-parsing capabilities provided a novel approach to identify potential student organizational tasks and deliverables. The findings suggest initial skepticism toward the tool with a limited adoption within the studied context, however, they also highlight the potential for AI-driven tools to provide students and professors valuable insights and formative support.

Problem

• Developing AI feedback for engineering student reports
• Overcoming initial adoption barriers for educational tools
• Improving report completeness through automated analysis

Innovation

• LLM-powered tool provides personalized feedback on drafts
• Task-parsing identifies student organizational tasks and deliverables
• AI-driven approach improves report completeness and quality