On the Challenges of Fuzzing Techniques via Large Language Models

2024-02-01

AI Summary

This paper addresses the low automation level and poor test case quality in software security fuzzing. It presents the first systematic survey on large language model (LLM)-empowered fuzzing, innovatively unifying three technical pathways: LLM-assisted fuzzing, traditional fuzzing enhancement via LLMs, and native LLM-based test case generation. Employing bibliometric analysis and systematic literature review, the study establishes a classification framework covering technical principles, implementation paradigms, and deployment feasibility. It identifies critical bottlenecks—including limited generalizability, reliability concerns, and scalability challenges—in current approaches. The work clarifies the technological evolution and core challenges of LLM-enhanced fuzzing, providing both theoretical foundations and practical guidelines for automated vulnerability detection. By bridging a significant gap, this survey constitutes the first comprehensive, cross-disciplinary synthesis of LLMs and fuzz testing research.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

In the modern era where software plays a pivotal role, software security and vulnerability analysis are essential for secure software development. Fuzzing test, as an efficient and traditional software testing method, has been widely adopted across various domains. Meanwhile, the rapid development in Large Language Models (LLMs) has facilitated their application in the field of software testing, demonstrating remarkable performance. As existing fuzzing test techniques are not fully automated and software vulnerabilities continue to evolve, there is a growing interest in leveraging large language models to generate fuzzing test. In this paper, we present a systematic overview of the developments that utilize large language models for the fuzzing test. To our best knowledge, this is the first work that covers the intersection of three areas, including LLMs, fuzzing test, and fuzzing test generated based on LLMs. A statistical analysis and discussion of the literature are conducted by summarizing the state-of-the-art methods up to date of the submission. Our work also investigates the potential for widespread deployment and application of fuzzing test techniques generated by LLMs in the future, highlighting their promise for advancing automated software testing practices.

Problem

• Automating fuzzing tests using large language models
• Addressing evolving software vulnerabilities with LLMs
• Exploring LLM-generated fuzzing for widespread deployment

Innovation

• Leveraging LLMs for automated fuzzing test generation
• Intersecting LLMs, fuzzing tests, and LLM-based generation
• Statistical analysis of state-of-the-art LLM fuzzing methods