LLM-Assisted Static Analysis for Detecting Security Vulnerabilities

2024-05-27

arXiv.org

AI Summary

Existing program analysis tools rely on manually defined taint specifications, limiting their scalability to repository-wide security vulnerability detection; while large language models (LLMs) excel at code generation, they lack cross-file, context-sensitive reasoning capabilities. This paper proposes IRIS, a neuro-symbolic framework that pioneers LLM-driven automatic taint specification inference, integrating GPT-4 with static analysis to enable fully automated, rule-free repository-scale taint tracking. IRIS supports context-sensitive, cross-module and cross-file analysis. Evaluated on CWE-Bench-Java, IRIS detects 55 vulnerabilities—28 more than CodeQL—with a 5-percentage-point reduction in false positives and identifies four previously undetected vulnerabilities missed by prior tools. The implementation is open-sourced.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Software is prone to security vulnerabilities. Program analysis tools to detect them have limited effectiveness in practice due to their reliance on human labeled specifications. Large language models (or LLMs) have shown impressive code generation capabilities but they cannot do complex reasoning over code to detect such vulnerabilities especially since this task requires whole-repository analysis. We propose IRIS, a neuro-symbolic approach that systematically combines LLMs with static analysis to perform whole-repository reasoning for security vulnerability detection. Specifically, IRIS leverages LLMs to infer taint specifications and perform contextual analysis, alleviating needs for human specifications and inspection. For evaluation, we curate a new dataset, CWE-Bench-Java, comprising 120 manually validated security vulnerabilities in real-world Java projects. A state-of-the-art static analysis tool CodeQL detects only 27 of these vulnerabilities whereas IRIS with GPT-4 detects 55 (+28) and improves upon CodeQL's average false discovery rate by 5% points. Furthermore, IRIS identifies 4 previously unknown vulnerabilities which cannot be found by existing tools. IRIS is available publicly at https://github.com/iris-sast/iris.

Problem

• Detects security vulnerabilities in software using LLMs and static analysis
• Reduces reliance on human-labeled specifications for vulnerability detection
• Improves detection rates and reduces false positives compared to existing tools

Innovation

• Combines LLMs with static analysis
• Infers taint specifications using LLMs
• Improves vulnerability detection accuracy