WaveformQA: Benchmarking LLM Temporal Reasoning on Digital Waveforms

2026-07-22

AI Summary

This work addresses the lack of systematic evaluation of large language models’ (LLMs’) temporal reasoning capabilities on digital waveforms, as existing benchmarks predominantly focus on hardware description language generation while neglecting complex timing and multi-signal relational reasoning. To bridge this gap, we introduce WaveformQA—an open-source question-answering benchmark comprising 360 programmatically generated questions derived from real-world open-source hardware designs, spanning eight progressively challenging temporal reasoning tasks. We innovatively replace conventional VCD waveform representations with an event-time JSON format, which substantially improves LLM inference accuracy and enables a standardized evaluation framework. Experimental results demonstrate that state-of-the-art LLMs perform adequately on simple queries but exhibit significant limitations in complex, multi-step or multi-signal scenarios; our proposed JSON representation effectively alleviates context-length and parsing bottlenecks, facilitating flexible extension to diverse question types and waveform sources.

Abstract

Large Language Models (LLMs) have demonstrated strong capabilities in code generation and reasoning, yet their ability to perform temporal reasoning over digital waveform data remains largely unexplored. Although reasoning over digital waveforms is a critical bottleneck in design verification, existing benchmarks primarily evaluate hardware description language (HDL) code generation and use waveforms only as supplementary context. This paper presents WaveformQA, an open-source question-answering benchmark for evaluating LLM temporal reasoning over digital waveforms. The benchmark comprises 360 questions with programmatically generated ground truths across eight categories of varying difficulty, including questions targeting multi-signal correlation and event ordering. Waveforms are generated from open-source design implementations, ensuring reproducibility and grounding the benchmark in real hardware behavior. Evaluation of frontier LLMs reveals that while models achieve reasonable accuracy on simple queries, performance degrades due to context window limitations and reasoning difficulties on complex temporal and multi-step questions. In addition, we show that an event-time JSON representation of waveforms improves LLM reasoning accuracy versus the standardized value change dump (VCD) format. The open-source framework supports extending to new question categories and importing new waveform sources, enabling researchers to rapidly prototype temporal reasoning experiments.

Problem

• temporal reasoning
• digital waveforms
• large language models
• design verification
• benchmarking

Innovation

• temporal reasoning
• digital waveforms
• LLM benchmarking
• event-time JSON
• hardware verification