MultiFuzz: A Dense Retrieval-based Multi-Agent System for Network Protocol Fuzzing

2025-08-19

AI Summary

Traditional protocol fuzzing suffers from weak semantic understanding and rigid seed mutation, while existing LLM-enhanced approaches (e.g., ChatAFL) are prone to hallucination and overreliance on inaccurate protocol knowledge. To address these issues, this paper proposes a dense-retrieval–based multi-agent fuzzing framework. It integrates RAG-augmented generation, RFC document chunking with embedding, and dense retrieval to build a semantic-aware contextual retrieval module; further, it employs specialized, functionally partitioned agents that collaboratively optimize message mutation via tool-augmented reasoning and chain-of-thought inference. Our key innovation lies in shifting protocol semantic understanding from “black-box prompting” to a closed-loop “retrieve–reason–verify” paradigm, enabling dynamic, adaptive state-space exploration. Evaluation on the RTSP protocol demonstrates significant improvements in branch coverage over NSFuzz, AFLNet, and ChatAFL, and discovers previously unknown deep-state transition paths for the first time.

Technology Category

• Multiagent Systems: Agent Communication
• Search and Optimization: Sampling/Simulation-based Search
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Security and Privacy: Large-scale security measurements

Abstract

Traditional protocol fuzzing techniques, such as those employed by AFL-based systems, often lack effectiveness due to a limited semantic understanding of complex protocol grammars and rigid seed mutation strategies. Recent works, such as ChatAFL, have integrated Large Language Models (LLMs) to guide protocol fuzzing and address these limitations, pushing protocol fuzzers to wider exploration of the protocol state space. But ChatAFL still faces issues like unreliable output, LLM hallucinations, and assumptions of LLM knowledge about protocol specifications. This paper introduces MultiFuzz, a novel dense retrieval-based multi-agent system designed to overcome these limitations by integrating semantic-aware context retrieval, specialized agents, and structured tool-assisted reasoning. MultiFuzz utilizes agentic chunks of protocol documentation (RFC Documents) to build embeddings in a vector database for a retrieval-augmented generation (RAG) pipeline, enabling agents to generate more reliable and structured outputs, enhancing the fuzzer in mutating protocol messages with enhanced state coverage and adherence to syntactic constraints. The framework decomposes the fuzzing process into modular groups of agents that collaborate through chain-of-thought reasoning to dynamically adapt fuzzing strategies based on the retrieved contextual knowledge. Experimental evaluations on the Real-Time Streaming Protocol (RTSP) demonstrate that MultiFuzz significantly improves branch coverage and explores deeper protocol states and transitions over state-of-the-art (SOTA) fuzzers such as NSFuzz, AFLNet, and ChatAFL. By combining dense retrieval, agentic coordination, and language model reasoning, MultiFuzz establishes a new paradigm in autonomous protocol fuzzing, offering a scalable and extensible foundation for future research in intelligent agentic-based fuzzing systems.

Problem

• Addresses unreliable LLM outputs and hallucinations in protocol fuzzing
• Overcomes limited semantic understanding of complex protocol grammars
• Enhances state coverage and adherence to syntactic constraints

Innovation

• Dense retrieval-based multi-agent system
• Semantic-aware context retrieval from RFCs
• Structured tool-assisted reasoning for mutations