Improving Automated Secure Code Reviews: A Synthetic Dataset for Code Vulnerability Flaws

2025-04-22

AI Summary

Current AI-based code review models exhibit limited performance in detecting security-critical vulnerabilities, primarily due to severe scarcity of vulnerability-labeled samples in real-world review data. To address this, we propose the first synthesis methodology for constructing code-vulnerability-oriented datasets: leveraging large language models (LLMs), it integrates fine-grained code-diff analysis, commit-level semantic understanding, and security-pattern-informed prompt engineering to systematically generate high-fidelity, interpretable, vulnerability-directed review comments. Unlike conventional data synthesis approaches, our method pioneers *vulnerability-driven* comment generation. Extensive experiments demonstrate that the synthesized data significantly improves three state-of-the-art review models—boosting average accuracy by 12.7% on both vulnerability detection and comment generation tasks, and increasing security coverage by 34.2%. These results empirically validate the effectiveness and necessity of vulnerability-specific synthetic data for training security-enhanced AI review systems.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Automation of code reviews using AI models has garnered substantial attention in the software engineering community as a strategy to reduce the cost and effort associated with traditional peer review processes. These models are typically trained on extensive datasets of real-world code reviews that address diverse software development concerns, including testing, refactoring, bug fixes, performance optimization, and maintainability improvements. However, a notable limitation of these datasets is the under representation of code vulnerabilities, critical flaws that pose significant security risks, with security-focused reviews comprising a small fraction of the data. This scarcity of vulnerability-specific data restricts the effectiveness of AI models in identifying and commenting on security-critical code. To address this issue, we propose the creation of a synthetic dataset consisting of vulnerability-focused reviews that specifically comment on security flaws. Our approach leverages Large Language Models (LLMs) to generate human-like code review comments for vulnerabilities, using insights derived from code differences and commit messages. To evaluate the usefulness of the generated synthetic dataset, we plan to use it to fine-tune three existing code review models. We anticipate that the synthetic dataset will improve the performance of the original code review models.

Problem

• Addressing underrepresentation of code vulnerabilities in AI training datasets
• Enhancing AI models' ability to identify security-critical code flaws
• Creating synthetic vulnerability-focused reviews using Large Language Models

Innovation

• Synthetic dataset for vulnerability-focused reviews
• LLMs generate human-like security flaw comments
• Fine-tune models with synthetic data for improvement