🤖 AI Summary
Traditional model-based testing (MBT) for protocols relies on manual RFC parsing to construct behavioral models, resulting in high modeling overhead and hindering practical adoption. This paper proposes an LLM-driven automated black-box testing framework that extracts protocol behavioral specifications from unstructured sources—including RFCs and technical blogs—using large language models, and integrates symbolic execution to generate semantically complete test cases. It introduces the first unified approach jointly performing behavioral modeling and oracle synthesis, enabling semantic-level protocol testing without manual modeling. Evaluated on DNS, the framework uncovered 26 real-world defects across 10 mainstream DNS implementations, including 11 previously unknown vulnerabilities. This significantly advances automation in protocol conformance testing and enhances vulnerability detection capability.
📝 Abstract
We present oracle-based testing a new technique for automatic black-box testing of network protocol implementations. Oracle-based testing leverages recent advances in LLMs to build rich models of intended protocol behavior from knowledge embedded in RFCs, blogs, forums, and other natural language sources. From these models it systematically derives exhaustive test cases using symbolic program execution. We realize oracle-based testing through Eywa, a novel protocol testing framework implemented in Python. To demonstrate Eywa's effectiveness, we show its use through an extensive case study of the DNS protocol. Despite requiring minimal effort, applying Eywa to the DNS resulting in the discovery of 26 unique bugs across ten widely used DNS implementations, including 11 new bugs that were previously undiscovered despite elaborate prior testing with manually crafted models.
