RTL-BenchLS: A Large-Scale Benchmark for RTL Reasoning and Generation with Large Language Models

2026-06-07

AI Summary

Existing RTL benchmarks are limited in scale and task diversity, making it difficult to accurately assess the capabilities of large language models in hardware design automation. This work introduces a large-scale benchmark comprising over 10,000 formally verified Verilog designs and proposes three novel tasks that integrate reasoning and generation. To overcome the scarcity of high-quality labeled data, it pioneers self-supervised round-trip and masked-content inference tasks. All results are automatically validated through formal equivalence checking, eliminating the need for manual testbenches. Experimental results reveal that state-of-the-art models achieve only 23%, 28%, and 12% accuracy on these new tasks—significantly lower than their performance on existing benchmarks—highlighting the challenge posed by this benchmark and its potential to drive future model advancements.

Abstract

LLM-based RTL generation and reasoning is a promising direction for hardware design automation. High-quality benchmarks are critical infrastructure for tracking progress in this direction. However, existing RTL benchmarks face inherent limitations in both scale and task scope. The designs they cover are typically small and simple, and the tasks focus almost entirely on specification-to-RTL generation. Frontier models' performance already saturates on the existing benchmarks. Scaling these benchmarks up is fundamentally difficult because aligned labels are required for benchmarking, such as specifications and testbenches. Such aligned high-quality data are rarely available for real-world designs. We introduce RTL-BenchLS, a large-scale benchmark addressing both limitations above. It contains over 10,000 formally verified Verilog designs, covering substantially larger and more complex designs than existing benchmarks. Beyond specification-to-RTL generation, we propose three novel tasks that jointly evaluate reasoning and generation: round-trip reasoning, masked-content reasoning, and repository-issue reasoning. The first two are self-supervised, which directly resolves the scaling bottleneck. All tasks are verified through formal equivalence checking without any manual testbenches. We evaluate eight LLMs on RTL-BenchLS. Even the best model reaches only 23% on natural-language round-trip reasoning, 28% on masked-content reasoning, and 12% on repository-issue fixing. RTL-BenchLS is substantially more challenging than existing benchmarks. It leaves ample room for future improvement and offers guidance for developing LLM-based methods for hardware design.

Problem

• RTL generation
• hardware design automation
• large language models
• benchmarking
• formal verification

Innovation

• RTL generation
• large-scale benchmark
• self-supervised reasoning
• formal verification
• LLM for hardware design