🤖 AI Summary
Current code generation evaluation benchmarks overemphasize correctness metrics while neglecting code quality and practical usability. This work proposes a tripartite evaluation framework that integrates complex project-based benchmarking, static code quality analysis, and structured developer reviews, thereby systematically incorporating real-world developer feedback into the assessment of large language models for code generation for the first time. Leveraging a tree-fold evaluation structure and multi-tiered computer science project benchmarks, experiments on GPT-4.1, DeepSeek-V3-0324, and Claude Opus 4 demonstrate that developer reviews effectively uncover critical production-level issues related to maintainability, readability, and engineering conventions, substantially addressing the limitations of traditional correctness-focused evaluations.
📝 Abstract
Code generation is one of the tasks for which the use of Large Language Models is widely adopted and highly successful. Given this popularity, there are many benchmarks dedicated to code generation that can help select the best model. However, they primarily focus on measuring solution correctness, leaving other aspects, such as code quality and usability, behind. This paper aims to describe a custom tree-fold evaluation methodology for code generated by Large Language Models that bridges this gap. The methodology includes a dedicated correctness benchmark based on a complex multi-level computer science project, code quality verification, and a survey of developers' opinions on generated code samples gathered through a structured code-review process. The proposed methodology's usage and usefulness are demonstrated by evaluating and comparing three general-purpose Large Language Models: GPT-4.1, DeepSeek-V3-0324, and Claude Opus 4. The results show that reviews gathered from developers can yield many new findings, especially those related to the code being in a production-ready state, that would not be possible to obtain using the standard correctness-focused benchmark approach.