Rubric Is All You Need: Enhancing LLM-based Code Evaluation With Question-Specific Rubrics

2025-03-31

AI Summary

This work addresses the weak logical reasoning and insufficient semantic alignment—despite strong generalizability—in LLM-based code assessment for educational settings. We propose a problem-customized multi-agent evaluation framework. Our key contributions are: (1) the first problem-aware method for automatic generation of fine-grained, context-specific scoring rubrics; (2) a novel metric, “Leniency”, quantifying evaluator permissiveness to enhance interpretability and controllability of strictness; and (3) the first real-world, education-oriented benchmark comprising 150+ data structures & algorithms (DSA) and 80+ object-oriented programming (OOP) student submissions. Experiments show our approach improves Spearman correlation with human judgments on logical correctness by 27.4% over baselines and reduces deviation from expert strictness ratings by 41%. The framework significantly enhances alignment with pedagogical objectives and feedback accuracy.

Technology Category

• Machine Learning: Evaluation and Analysis
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Search and Optimization: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Since the disruption in LLM technology brought about by the release of GPT-3 and ChatGPT, LLMs have shown remarkable promise in programming-related tasks. While code generation remains a popular field of research, code evaluation using LLMs remains a problem with no conclusive solution. In this paper, we focus on LLM-based code evaluation and attempt to fill in the existing gaps. We propose multi-agentic novel approaches using question-specific rubrics tailored to the problem statement, arguing that these perform better for logical assessment than the existing approaches that use question-agnostic rubrics. To address the lack of suitable evaluation datasets, we introduce two datasets: a Data Structures and Algorithms dataset containing 150 student submissions from a popular Data Structures and Algorithms practice website, and an Object Oriented Programming dataset comprising 80 student submissions from undergraduate computer science courses. In addition to using standard metrics (Spearman Correlation, Cohen's Kappa), we additionally propose a new metric called as Leniency, which quantifies evaluation strictness relative to expert assessment. Our comprehensive analysis demonstrates that question-specific rubrics significantly enhance logical assessment of code in educational settings, providing better feedback aligned with instructional goals beyond mere syntactic correctness.

Problem

• Enhancing LLM-based code evaluation with question-specific rubrics
• Addressing lack of suitable datasets for code evaluation
• Improving logical assessment in educational settings beyond syntax

Innovation

• Uses question-specific rubrics for logical assessment
• Introduces two new code evaluation datasets
• Proposes Leniency metric for evaluation strictness