What Bugs Do Prolog Students Write? An Empirical Taxonomy and Data-Driven Mutation Framework

2026-07-23

AI Summary

This study addresses the misalignment between errors generated by existing Prolog mutation testing tools and the actual distribution of student programming mistakes, which limits their utility for effective pedagogical feedback. Drawing on 7,201 homework submissions from 265 undergraduate students and 200 repair commits, the authors present the first empirically grounded, fine-grained taxonomy of student errors in Prolog. Building upon this taxonomy, they propose LogMorph, a data-driven mutation tool that synthesizes realistic program faults by leveraging abstract syntax tree analysis, mutation operators weighted by empirical error frequencies, and an SMT-solver-enhanced code synthesis mechanism. Evaluation on 16,000 generated mutants demonstrates that the induced error distributions align closely with real student errors—deviating by no more than two percentage points across most categories—thereby validating the approach’s effectiveness and novelty.

Abstract

Automated feedback tools for logic programming education depend on realistic bug datasets that reflect the mistakes students actually make. However, existing mutation testing frameworks for Prolog treat all mutations as equally likely, producing synthetic faults that diverge from classroom reality. We present an empirical study of 7,201 Prolog submissions from 265 undergraduate students, from which we derive a fine-grained taxonomy of student bugs through manual classification of 200 bug-fixing submissions. Guided by this taxonomy, we develop LogMorph, a data-driven mutation tool whose 17 operators are weighted according to the observed error distribution. LogMorph enumerates valid mutation sites on the abstract syntax tree, samples operators proportionally, injects faults, delegating to an SMT-based synthesizer when new code fragments are needed, and validates each mutant against a reference test suite. An evaluation of 16,000 generated mutants shows that the synthetic error distribution closely matches the student distribution, with most bug categories agreeing to within two percentage points. We identify cut-related mutations and synthesizer-generated code as the main sources of residual divergence, and outline how combining the SMT back-end with a language model fine-tuned on student code can further improve realism.

Problem

• Prolog
• student bugs
• mutation testing
• error taxonomy
• automated feedback

Innovation

• data-driven mutation
• empirical bug taxonomy
• LogMorph
• SMT-based synthesis
• Prolog education