Unreliable in Practice? A Comprehensive Study of Errors in LLM-Generated Code

2026-08-01

AI Summary

This study addresses the lack of systematic understanding of error types in code generated by large language models (LLMs), which hinders reliable evaluation and mitigation strategies. The authors construct the first large-scale annotated dataset comprising 86,726 erroneous code samples across seven prominent LLMs and four compiled languages. Leveraging LLM-assisted error attribution, manual verification, and combined static and dynamic analysis, they conduct a comprehensive cross-model, cross-language, and cross-task-difficulty investigation of compilation and runtime errors. Their findings reveal that even the largest models frequently commit elementary mistakes, commonly exhibit insufficient input validation and memory safety vulnerabilities, and display error distributions significantly shaped by both model architecture and programming language—providing critical empirical foundations for improving the reliability of LLM-generated code.

Abstract

Large Language Models (LLMs) are being widely used for coding, with reports indicating that AI now generates an increasing share of production code. Studies show that LLMs can significantly improve developer productivity, yet they still struggle with more complex coding tasks. Just as understanding error modes in human-written code has been central to improving software quality, identifying and characterizing the errors in LLM-generated code is critical for setting realistic expectations and designing mitigation strategies. Prior research has been limited in scope, often focusing on a single language, a small number of problems, or a limited selection of models. As a result, there is still no comprehensive understanding of which errors are common and which are specific to certain models or languages. To address these gaps and develop a deeper understanding of the quality of LLM-generated code, we analyzed a corpus of 86,726 code samples that contained compilation or runtime errors. These samples were generated by seven LLMs across four compiled languages. We classified errors by their underlying causes using an LLM, manually validated these classifications, and performed a comparative analysis. This labeled data is then used to measure error prevalence by model, language, and problem difficulty, to identify common error patterns. Results show that, although error types vary strongly across languages and models, even the largest models frequently make simple mistakes. We also observe that generated code often omits basic input validation or memory-safety checks, which can lead to overflows, resource exhaustion, or other reliability/security issues.

Problem

• LLM-generated code
• error analysis
• code reliability
• software quality
• compilation errors

Innovation

• LLM-generated code
• error analysis
• code reliability
• cross-model evaluation
• memory safety