QASM-Eval: A Dataset to Train and Evaluate LLMs on OpenQASM-3 Beyond Quantum Circuits

2026-04-28

arXiv.org

AI Summary

This study addresses the bottleneck of scarce training and evaluation data for large language models in hardware-level OpenQASM 3 programming, such as quantum error correction and pulse control. To this end, we construct the first comprehensive benchmark dataset encompassing classical logic, temporal scheduling, and pulse-level control. Methodologically, we employ an expert-curated data construction strategy and design an automated verifier integrating syntax checking, measurement distribution validation, and timing verification. Experimental results demonstrate that fine-tuning Llama-3-70B achieves a pass@1 accuracy of 61.17%, approaching the few-shot performance of GPT-5.6-Terra while significantly outperforming zero-shot baselines. This work effectively bridges the critical gap in evaluating low-level quantum programming capabilities.

Abstract

Quantum computing remains in the Noisy Intermediate-Scale Quantum (NISQ) era, where the performance is highly constrained to noise. Addressing the limitation often requires hardware-facing capabilities beyond gate-sequence circuit specification, including mid-circuit measurement and classical feedback for quantum error correction (QEC), precise timing control for dynamical decoupling (DD), and pulse-level waveform access for calibration. OpenQASM-3 was introduced to expose exactly these capabilities, providing a hardware-level programming interface. However, despite the rapid progress of large language models in code generation, there is still no dataset specifically designed to train and evaluate LLMs on OpenQASM-3 programs that involve its advanced hardware-oriented features. To address this gap, we introduce QASM-Eval, the first comprehensive dataset designed to train and evaluate LLMs on OpenQASM-3. Rather than focusing on quantum algorithm design or reasoning, QASM-Eval explicitly targets the language's hardware-facing features. QASM-Eval comprises an expert-verified test set of 100 tasks and a training set of 4,000 tasks, systematically covering classical logic, timing scheduling, pulse control, and complex real-world workflows. To automatically validate generated programs, we check syntax, quantum states and program timeline using an extended verifier. Our evaluation reveals that while state-of-the-art LLMs struggle heavily in OpenQASM-3 coding tasks, targeted fine-tuning on QASM-Eval yields significant gains. QASM-Eval provides a crucial benchmark and training foundation to accelerate the development of reliable LLM assistants for hardware-facing quantum programming in NISQ era. Data and code: https://github.com/fuzhenxiao/QASM-Eval

Problem

• OpenQASM 3
• Large Language Models
• Quantum Computing
• Code Generation
• Benchmark Dataset

Innovation

• OpenQASM 3
• QASM-Eval
• Large Language Models
• Quantum Code Generation
• Benchmark Dataset