Exploring LLMs Impact on Student-Created User Stories and Acceptance Testing in Software Development

2025-02-04

AI Summary

This study investigates how large language models (LLMs) influence undergraduate students’ ability to transform user feedback into user stories adhering to the INVEST criteria (Independent, Negotiable, Valuable, Estimable, Small, Testable). Method: We employed an instruction-tuned LLM within a mixed-methods framework—combining qualitative coding and quantitative evaluation—to compare an experimental group (LLM-assisted story writing) against a control group (fully manual). Contribution/Results: Results demonstrate that LLM assistance significantly improves the completeness and testability of acceptance criteria—the first empirical evidence of LLMs’ tangible benefit in acceptance test generation. However, students using LLMs produced user stories with excessive scope (i.e., poor granularity control), whereas the manual group exhibited superior story sizing. The study thus reveals a dual-edged impact of LLMs in software requirements engineering education, providing critical empirical evidence and actionable insights for AI-integrated engineering pedagogy.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: User Experience and Usability
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

In Agile software development methodology, a user story describes a new feature or functionality from an end user's perspective. The user story details may also incorporate acceptance testing criteria, which can be developed through negotiation with users. When creating stories from user feedback, the software engineer may maximize their usefulness by considering story attributes, including scope, independence, negotiability, and testability. This study investigates how LLMs (large language models), with guided instructions, affect undergraduate software engineering students' ability to transform user feedback into user stories. Students, working individually, were asked to analyze user feedback comments, appropriately group related items, and create user stories following the principles of INVEST, a framework for assessing user stories. We found that LLMs help students develop valuable stories with well-defined acceptance criteria. However, students tend to perform better without LLMs when creating user stories with an appropriate scope.

Problem

• Impact of LLMs on student-created user stories
• Transforming user feedback into effective user stories
• Role of LLMs in defining acceptance criteria

Innovation

• LLMs enhance user story creation
• Guided instructions optimize feedback analysis
• Students outperform without LLMs in scope