Utilizing Precise and Complete Code Context to Guide LLM in Automatic False Positive Mitigation

2024-11-05

arXiv.org

AI Summary

Static Application Security Testing (SAST) tools suffer from high false positive rates, severely impeding developer productivity. Existing LLM-based false positive mitigation approaches are limited by coarse-grained context extraction and inability to capture cross-file dependencies, resulting in suboptimal discrimination accuracy. This paper proposes LLM4FPM, a lightweight framework that introduces two novel techniques: eCPG-Slicer and FARF. These enable linear-time, eCPG-driven, line-level precise context slicing and cross-file relevance identification. By synergistically integrating static analysis with open-source lightweight LLMs, LLM4FPM balances contextual completeness and computational efficiency. Evaluated on the Juliet dataset, it achieves an F1 score of 99.1% and 86% accuracy on D2A (Defect-to-Alert) labeling. Each warning is processed at an average cost of $0.384 and latency of 4.7 seconds—substantially reducing manual review overhead.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)Planning, Routing, and Scheduling: Planning with Language Models

Application Category

Economics, Online Markets and Human Computation: Cost models of using LLMs in production systemsGraph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphsSearch and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Static Application Security Testing (SAST) tools are critical to software quality, identifying potential code issues early in development. However, they often produce false positive warnings that require manual review, slowing down development. Thus, automating false positive mitigation (FPM) is essential. The advent of Large Language Models (LLMs), with their strong abilities in natural language and code understanding, offers promising avenues for FPM. Yet current LLM-based FPM method faces two major limitations: 1. The warning-related code snippets extracted are overly broad and cluttered with irrelevant control/data flows, reducing precision; 2. Critical code contexts are missing, leading to incomplete representations that can mislead LLMs and cause inaccurate assessments. To overcome these limitations, we propose LLM4FPM , a lightweight and efficient false positive mitigation framework. It features eCPG-Slicer, which builds an extended code property graph (eCPG) to extract precise line-level code contexts for warnings. Furthermore, the integrated FARF algorithm builds a file reference graph to identify all files that are relevant to warnings in linear time. This enables eCPG-Slicer to obtain rich contextual information without resorting to expensive whole-program analysis. LLM4FPM outperforms the existing method on the Juliet dataset (F1>99% across various Common Weakness Enumerations) and improves label accuracy on the D2A dataset to 86%. By leveraging a lightweight open-source LLM, LLM4FPM can significantly save inspection costs up to $2758 per run ($0.384 per warning) on Juliet with an average inspection time of 4.7s per warning. Moreover, real-world tests on popular C/C++ projects demonstrate its practicality.

Problem

• Automating false positive mitigation in SAST tools using LLMs
• Overcoming imprecise and incomplete code context in LLM-based FPM
• Enhancing accuracy and efficiency in static security warning analysis

Innovation

• Uses eCPG-Slicer for precise code context extraction
• Integrates FARF algorithm for file relevance identification
• Leverages lightweight LLM to reduce inspection costs