Detect Repair Verify for Securing LLM Generated Code: A Multi-Language Empirical Study

2026-02-28

AI Summary

This work addresses the lack of security and verifiability in large language models for project-level code generation by proposing and evaluating an end-to-end Detect–Repair–Verify (DRV) workflow tailored for multilingual web applications. The approach generates executable code at three granularities—project, requirement, and function—integrating static and dynamic analysis, automated repair, and test-driven verification. Under unified resource constraints, the study systematically compares generative, single-round, and iterative variants of DRV. It introduces the first project-level benchmark for secure code generation that supports multiple prompting granularities, enabling a comprehensive evaluation of DRV’s efficacy. The findings reveal limitations in using vulnerability reports to guide repairs and identify common post-repair failure modes such as regressions and semantic drift. Experimental results demonstrate that the iterative DRV variant significantly enhances security while preserving functional correctness.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large language models are increasingly used to produce runnable software. In practice, security is often addressed through a Detect--Repair--Verify (DRV) loop that detects issues, applies fixes, and verifies the result. This work studies such a workflow for project-level artifacts and addresses four gaps: L1, the lack of project-level benchmarks with executable function and security tests; L2, limited evidence on pipeline-level effectiveness beyond studying detection or repair alone; L3, unclear reliability of detection reports as repair guidance; and L4, uncertain repair trustworthiness and side effects under verification. A new benchmark dataset\footnote{https://github.com/Hahappyppy2024/EmpricalVDR} is introduced, consisting of runnable web-application projects paired with functional tests and targeted security tests, and supporting three prompt granularities at the project, requirement, and function level. The evaluation compares generation-only, single-pass DRV, and bounded iterative DRV variants under comparable budget constraints. Outcomes are measured by secure and correct yield using test-grounded verification, and intermediate artifacts are analyzed to assess report actionability and post-repair failure modes such as regressions, semantic drift, and newly introduced security issues.

Problem

• Large Language Models
• Code Security
• Detect-Repair-Verify
• Project-level Benchmark
• Empirical Study

Innovation

• Detect-Repair-Verify
• project-level benchmark
• LLM-generated code security
• test-grounded verification
• repair trustworthiness