🤖 AI Summary
Computer science students frequently exhibit persistent misconceptions about core testing concepts—particularly decision coverage and exception handling—leading to superficial test modifications that fail to improve code coverage. This study employs an automated testing feedback tool to systematically analyze multi-version student submissions and corresponding coverage changes, providing the first empirical identification of these two conceptual misunderstandings and their strong association with ineffective modification behaviors. Results show that over 70% of student test modifications yield no statistically significant coverage improvement, with errors predominantly occurring in branch decision logic and omitted exception-handling paths. The work reveals critical cognitive bottlenecks in testing education and delivers empirically grounded, actionable insights for designing targeted pedagogical interventions and intelligent, context-aware feedback tools.
📝 Abstract
Effective software testing is critical for producing reliable and secure software, yet many computer science students struggle to master the foundational concepts required to construct comprehensive test suites. While automated feedback tools are widely used to support student learning, it remains unclear which testing concepts are most frequently misunderstood and how these misunderstandings are reflected in students' test suite revisions. This study examines the specific testing concepts that lead students to make ineffective changes, those that fail to improve code coverage, during test suite development. Leveraging an automated feedback tool in a senior-level software testing course, we analyzed student submissions from two assignments to identify prevalent conceptual gaps and patterns of unproductive modification. Our results reveal that decision coverage and exception handling are persistent challenges, and that students most often make superficial or method-level changes that do not enhance coverage. These findings provide actionable insights for educators, researchers, and tool designers. By pinpointing the concepts that most often contribute to poor testing outcomes, we can refine feedback systems, target instruction to address persistent misconceptions, and more effectively support students in developing robust, maintainable test suites.