Effective Command-line Interface Fuzzing with Path-Aware Large Language Model Orchestration

2025-11-25

AI Summary

Existing CLI fuzzing techniques struggle to generate semantically rich command-line arguments and input files, resulting in insufficient coverage of deep target functions and low vulnerability detection rates. This paper proposes PILOT, a path-aware large language model (LLM) collaborative framework: it leverages static analysis to extract potential calling paths to target functions as contextual prompts for the LLM, then jointly mutates command-line options and input files in a path-directed manner by integrating dynamic execution feedback with LLM-based reasoning; iterative feedback further refines coverage over time. Evaluated on real-world software, PILOT achieves significantly higher code coverage than state-of-the-art approaches and discovers 51 zero-day vulnerabilities—41 confirmed by developers, 33 patched, and 3 assigned CVE identifiers.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Planning, Routing, and Scheduling: Planning with Language Models
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Command-line interface (CLI) fuzzing tests programs by mutating both command-line options and input file contents, thus enabling discovery of vulnerabilities that only manifest under specific option-input combinations. Prior works of CLI fuzzing face the challenges of generating semantics-rich option strings and input files, which cannot reach deeply embedded target functions. This often leads to a misdetection of such a deep vulnerability using existing CLI fuzzing techniques. In this paper, we design a novel Path-guided, Iterative LLM-Orchestrated Testing framework, called PILOT, to fuzz CLI applications. The key insight is to provide potential call paths to target functions as context to LLM so that it can better generate CLI option strings and input files. Then, PILOT iteratively repeats the process, and provides reached functions as additional context so that target functions are reached. Our evaluation on real-world CLI applications demonstrates that PILOT achieves higher coverage than state-of-the-art fuzzing approaches and discovers 51 zero-day vulnerabilities. We responsibly disclosed all the vulnerabilities to their developers and so far 41 have been confirmed by their developers with 33 being fixed and three assigned CVE identifiers.

Problem

• CLI fuzzing struggles to generate semantic-rich options and input files
• Existing techniques fail to reach deeply embedded target functions
• Deep vulnerabilities often remain undetected by current CLI fuzzing methods

Innovation

• Path-guided iterative LLM orchestration for CLI fuzzing
• Using call paths as context to generate CLI options
• Iteratively updates context with reached functions