Large Language Model-Driven Concolic Execution for Highly Structured Test Input Generation

2025-04-24

AI Summary

This work addresses the systematic testing challenge for parser programs, tackling key bottlenecks in existing concolic execution: blind path selection, constraint-solving failures on structured inputs, and heavy reliance on manually crafted seeds. To resolve these, we propose (1) ESCT—a syntax-aware structural coverage tree for guided path-constraint selection; (2) the first LLM-driven Solve-Complete constraint-solving paradigm, significantly improving solvability of complex syntactic constraints; and (3) a history-guided automatic seed evolution mechanism. We implement a prototype system built upon SymCC, integrating LLMs, path modeling, and syntax-aware solving. Evaluated on eight open-source libraries across XML, SQL, JavaScript, and JSON parsers, our approach achieves 14.15% and 14.31% higher line coverage than SymCC and Marco, respectively, and discovers six previously unknown vulnerabilities—all assigned CVE identifiers, four of which have been confirmed and patched.

Technology Category

• Constraint Satisfaction and Optimization: Solvers and Tools
• Natural Language Processing: Syntax — Tagging, Chunking & Parsing
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

How can we perform concolic execution to generate highly structured test inputs for systematically testing parsing programs? Existing concolic execution engines are significantly restricted by (1) input structure-agnostic path constraint selection, leading to the waste of testing effort or missing coverage; (2) limited constraint-solving capability, yielding many syntactically invalid test inputs; (3) reliance on manual acquisition of highly structured seed inputs, resulting in non-continuous testing. This paper proposes Cottontail, a new Large Language Model (LLM)-driven concolic execution engine, to mitigate the above limitations. A more complete program path representation, named Expressive Structural Coverage Tree (ESCT), is first constructed to select structure-aware path constraints. Later, an LLM-driven constraint solver based on a Solve-Complete paradigm is designed to solve the path constraints smartly to get test inputs that are not only satisfiable to the constraints but also valid to the input syntax. Finally, a history-guided seed acquisition is employed to obtain new highly structured test inputs either before testing starts or after testing is saturated. We implemented Cottontail on top of SymCC and evaluated eight extensively tested open-source libraries across four different formats (XML, SQL, JavaScript, and JSON). The experimental result is promising: it shows that Cottontail outperforms state-of-the-art approaches (SymCC and Marco) by 14.15% and 14.31% in terms of line coverage. Besides, Cottontail found 6 previously unknown vulnerabilities (six new CVEs have been assigned). We have reported these issues to developers, and 4 out of them have been fixed so far.

Problem

• Generate structured test inputs for parsing programs
• Improve constraint-solving to avoid invalid test inputs
• Automate acquisition of structured seed inputs

Innovation

• LLM-driven concolic execution engine
• Expressive Structural Coverage Tree (ESCT)
• Solve-Complete paradigm constraint solver