Secure Coding with AI, From Creation to Inspection

2025-04-29

AI Summary

This study systematically evaluates ChatGPT’s security behavior in realistic development interactions—specifically its propensity to generate vulnerable code and its capability to autonomously detect and remediate security flaws. Leveraging the DevGPT dataset, we integrate static analysis tools (Clang Static Analyzer, SonarQube), manual validation, and OpenAI API queries to quantify vulnerability origins in AI-generated C/C++/C# code for the first time: 10% of vulnerabilities stem from user prompts, while 22% are directly introduced by ChatGPT. Results show that AI-generated code exhibits a significantly higher vulnerability density than human-written code. Among 32 manually confirmed vulnerabilities, ChatGPT detected only 18 and successfully repaired 17; critically, its high-confidence incorrect responses risk misleading novice developers. The findings expose fundamental limitations of current large language models in security-critical software tasks, underscoring that automated static analysis and expert human review remain indispensable safeguards.

Technology Category

• Natural Language Processing: Safety and Robustness
• Computer Vision: Large Vision Models
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness

Application Category

• Security and Privacy: Large-scale security measurements
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

While prior studies have explored security in code generated by ChatGPT and other Large Language Models, they were conducted in controlled experimental settings and did not use code generated or provided from actual developer interactions. This paper not only examines the security of code generated by ChatGPT based on real developer interactions, curated in the DevGPT dataset, but also assesses ChatGPT's capability to find and fix these vulnerabilities. We analysed 1,586 C, C++, and C# code snippets using static scanners, which detected potential issues in 124 files. After manual analysis, we selected 26 files with 32 confirmed vulnerabilities for further investigation. We submitted these files to ChatGPT via the OpenAI API, asking it to detect security issues, identify the corresponding Common Weakness Enumeration numbers, and propose fixes. The responses and modified code were manually reviewed and re-scanned for vulnerabilities. ChatGPT successfully detected 18 out of 32 security issues and resolved 17 issues but failed to recognize or fix the remainder. Interestingly, only 10 vulnerabilities were resulted from the user prompts, while 22 were introduced by ChatGPT itself. We highlight for developers that code generated by ChatGPT is more likely to contain vulnerabilities compared to their own code. Furthermore, at times ChatGPT reports incorrect information with apparent confidence, which may mislead less experienced developers. Our findings confirm previous studies in demonstrating that ChatGPT is not sufficiently reliable for generating secure code nor identifying all vulnerabilities, highlighting the continuing importance of static scanners and manual review.

Problem

• Assess ChatGPT's ability to detect and fix vulnerabilities in real developer-generated code
• Compare vulnerability rates between developer-written and ChatGPT-generated code snippets
• Evaluate reliability of ChatGPT for secure coding without manual review tools

Innovation

• Analyzed ChatGPT code security in real developer interactions
• Used static scanners and manual review for vulnerability detection
• Assessed ChatGPT's ability to detect and fix vulnerabilities