Accelerating Accurate Assignment Authoring Using Solution-Generated Autograders

2026-08-06

AI Summary

This work addresses the limitations of traditional automated grading systems for programming assignments, which rely on manually crafted test cases that are labor-intensive, prone to missing edge cases, and hinder large-scale question bank development. The authors propose a reference-solution-centric approach to automatically generate grading logic by leveraging program analysis and dynamic testing techniques, eliminating the need for hand-written test cases. The method not only verifies functional correctness but also evaluates code quality. Implemented as the Questioner system, this approach has been deployed in Java and Kotlin environments and successfully supported nearly 800 programming problems and millions of student submissions over four years in CS1 courses, significantly improving problem authoring efficiency, grading accuracy, and system scalability.

Abstract

Students learning to program benefit from access to large numbers of practice problems. Autograders are commonly used to support programming questions by providing quick feedback on submissions. But authoring accurate autograders remains challenging. Autograders are frequently created by enumerating test cases--a tedious process that can produce inaccurate autograders that fail to correctly classify submissions. When authoring accurate autograders is slow, it is difficult to create large banks of practice problems to support beginning programmers. We present solution-generated autograding: a faster, more accurate, and more enjoyable way to create autograders. Our approach leverages a key difference between software testing and autograding: The question author can provide a solution. By starting with a solution, we can eliminate the need to manually enumerate test cases, validate the autograder's accuracy, and evaluate other aspects of submission code quality beyond behavioral correctness. We describe Questioner, an implementation of solution-generated autograding for Java and Kotlin, and share experiences from four years using Questioner to support a large CS1 course: authoring nearly 800 programming questions used by thousands of students to evaluate millions of submissions.

Problem

• autograding
• programming education
• test case authoring
• CS1
• automated assessment

Innovation

• solution-generated autograding
• autograder authoring
• programming education
• automated assessment
• test case generation