FirmReBugger: A Benchmark Framework for Monolithic Firmware Fuzzers

2026-01-22

AI Summary

This work addresses the lack of reliable, real-world vulnerability-based benchmarks for evaluating firmware fuzzers, which hinders objective assessment of code coverage and crash effectiveness. To this end, we propose FirmBench, an end-to-end evaluation framework that introduces a novel oracle mechanism requiring no modification to target binaries. The approach leverages C-syntax specifications to describe vulnerability states—unreached, reached, triggered, and detected—and integrates seed replay with a vulnerability state interpreter to enable automated, reproducible fuzzer performance analysis. Using a diverse benchmark suite comprising 313 real-world vulnerability oracles and extensive experiments totaling 10 CPU-years, FirmBench successfully evaluates nine state-of-the-art monolithic firmware fuzzers, demonstrating its effectiveness and practical utility.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization

Abstract

Monolithic Firmware is widespread. Unsurprisingly, fuzz testing firmware is an active research field with new advances addressing the unique challenges in the domain. However, understanding and evaluating improvements by deriving metrics such as code coverage and unique crashes are problematic, leading to a desire for a reliable bug-based benchmark. To address the need, we design and build FirmReBugger, a holistic framework for fairly assessing monolithic firmware fuzzers with a realistic, diverse, bug-based benchmark. FirmReBugger proposes using bug oracles--C syntax expressions of bug descriptors--with an interpreter to automate analysis and accurately report on bugs discovered, discriminating between states of detected, triggered, reached and not reached. Importantly, our idea of benchmarking does not modify the target binary and simply replays fuzzing seeds to isolate the benchmark implementation from the fuzzer while providing a simple means to extend with new bug oracles. Further, analyzing fuzzing roadblocks, we created FirmBench, a set of diverse, real-world binary targets with 313 software bug oracles. Incorporating our analysis of roadblocks challenging monolithic firmware fuzzing, the bench provides for rapid evaluation of future advances. We implement FirmReBugger in a FuzzBench-for-Firmware type service and use FirmBench to evaluate 9 state-of-the art monolithic firmware fuzzers in the style of a reproducibility study, using a 10 CPU-year effort, to report our findings.

Problem

• firmware fuzzing
• benchmark
• bug oracle
• monolithic firmware
• fuzzer evaluation

Innovation

• firmware fuzzing
• bug oracle
• benchmark framework
• monolithic firmware
• reproducible evaluation