microbenchmark design

Existing CPU benchmarks (e.g., SPEC CPU2017) lack explicit system configuration specifications, leading to performance interference from non-CPU components and severely undermining comparability, consistency, and reproducibility. Method: We propose a novel CPU performance evaluation paradigm grounded in the principle of “fully specified and valid configurations,” establishing a systematic modeling framework that spans the complete configuration space; we design an unbiased sampling strategy that uniformly weights all compliant configurations; and we replace point estimates with confidence intervals and associated confidence levels for performance reporting. Results: Experiments reveal up to 74.49× performance variation for the same CPU across compliant configurations. Our framework eliminates configuration ambiguity entirely, enabling fair cross-CPU comparisons and significantly improving consistency, reproducibility, and statistical rigor of benchmarking outcomes.

Accurately characterizing the timing behavior of heterogeneous memories (e.g., PL-side DRAM, BRAM) in embedded systems under configurable contention pressure remains challenging due to measurement noise and limitations of user-space tools. Method: This paper proposes MemScope, a kernel-level heterogeneous memory characterization framework. It introduces a novel kernel-level fine-grained memory control mechanism enabling direct physical memory mapping, dynamic inter-core interference modeling and mitigation, and tightly integrated support for multi-core scheduling coordination, cache coherency maintenance, and joint interrupt/I/O regulation. Contribution/Results: Evaluated on a Xilinx ZCU102 platform, MemScope achieves high consistency (<3% deviation) and nanosecond-level resolution in bandwidth and latency benchmarking across multiple memory types. By significantly suppressing measurement noise and overcoming the precision ceiling of user-mode tools, MemScope establishes a reliable infrastructure for real-time analysis of heterogeneous SoCs.

This work addresses the limitations of existing CPU benchmarks in accurately evaluating the performance of modern heterogeneous, multithreaded processors under diverse workloads. To this end, the authors present the SPEC CPU 2026 benchmark suite, developed through community collaboration and principled methodology, which introduces the Rolling-Round-Robin Rate approach to standardize the execution of heterogeneous multiprogrammed workloads. The suite incorporates newly designed multithreaded benchmarks exhibiting varied microarchitectural characteristics, selected and hardened through an open-source application curation process. Emphasizing workload diversity, portability, and long-term viability, SPEC CPU 2026 establishes a robust, representative, and authoritative standard for performance evaluation, thereby supporting next-generation computer architecture research.

This work addresses the growing mismatch between modern datacenter and AI workloads and traditional CPU benchmarks, which often fail to accurately capture performance bottlenecks. The study presents the first systematic microarchitectural analysis of SPEC CPU2026 across nine mainstream processors, leveraging cross-platform performance counters, clustering algorithms, and detailed case studies—such as page size effects and prefetching strategies—to construct a highly representative compact subset comprising only four to five programs. This subset preserves 96.4%–99.9% of the original benchmark suite’s behavioral characteristics. Furthermore, the authors introduce a polling-interleaved pattern to synthesize proxy workloads, reducing the IPC gap with real-world DCPerf workloads to just 13.7%, thereby significantly enhancing both the efficiency and representativeness of CPU performance evaluation.

This work addresses the lack of automated, efficient methods for evaluating how closely existing benchmarks resemble real-world high-performance computing (HPC) applications in terms of hardware performance characteristics. The authors propose a novel performance similarity metric based on hardware usage patterns, introducing for the first time two distinct classes of computational kernels that exhibit similar performance behavior. They develop a scalable, automated evaluation framework that integrates performance feature analysis, kernel classification, and cross-platform (CPU/GPU) similarity assessment. The effectiveness and practicality of this approach are demonstrated by accurately matching computational kernels from the Kripke proxy application to those in the RAJA Performance Suite, thereby validating the method’s capability to identify functionally analogous kernels across diverse hardware architectures.

This work addresses the lack of reproducible and fairly comparable benchmarks in quantum software testing, which has largely relied on small, hard-coded circuits that poorly reflect real-world development practices. To bridge this gap, the authors introduce Qolumbina—the first scalable benchmark suite for quantum software testing—systematically curated from 40 representative programs sourced from open-source repositories and rigorously refactored with standardized interfaces, test cases, and formal specifications. The study further proposes a novel taxonomy of testing characteristics specific to quantum programs and employs program complexity modeling to enable scalability analysis and systematic evaluation. Covering a diverse range of testing attributes, Qolumbina has already facilitated empirical studies on execution overhead and fault detection capability, revealing the critical influence of backend dependencies on the interpretation of testing outcomes.

Hardware benchmarking often suffers from limited verifiability, hindering reproducibility and auditability, and is vulnerable to silent computational errors. This work proposes the first end-to-end verifiable measurement logging system, which integrates transparent logs, probabilistic verification (e.g., Freivalds’ algorithm), and an adversarial-aware Fiat-Shamir challenge mechanism to bind all observations into an append-only hash chain via content-based hashing, enabling offline auditing. The system incorporates algebraic checks, floating-point error tolerance calibration, and di/dt power and thermal stress testing to effectively distinguish physical faults from malicious tampering. Experiments on Blackwell and Hopper GPUs demonstrate that the approach accurately characterizes residual noise floors across varying precisions and scales, detects no undetected silent errors, and thereby validates its effectiveness and robustness.

Existing methodologies—such as SPEC CPU2017, Design of Experiments (DoE), and Randomized Controlled Trials (RCTs)—struggle to accurately attribute overall system performance to individual hardware components due to their inability to effectively isolate component-level contributions, resulting in substantial evaluation variability (SPEC score deviations ranging from 12.16% to 436.80%). This work proposes a novel methodology that integrates controlled experimentation with a theoretical attribution model, enabling, for the first time, precise and stable attribution of system performance to specific hardware components. The proposed approach significantly outperforms conventional techniques, offering high cost-effectiveness while overcoming inherent limitations in component evaluation and system design present in current practices.

This work addresses the challenge of efficiently conducting “What-If” I/O performance analysis for large-scale HPC applications, which is hindered by the complex interplay among access patterns, middleware, and file systems. The authors propose FBench, the first flexible I/O benchmarking tool based on context-free grammars (CFGs), capable of generating or replaying I/O traces—captured via Recorder—in real time without modifying application code. FBench supports both POSIX and MPI-IO interfaces and enables configuration-driven exploration through JSON-defined optimization strategies. It faithfully reproduces real-world workloads such as IOR, HACC-IO, FLASH Sedov, and LAMMPS. Evaluations on Lustre reveal that collective I/O write bandwidth can be up to 30× lower than ideal, burst buffers improve non-collective write bandwidth by 1.5×, and performance gains of up to 8× are achievable in LAMMPS scenarios, significantly accelerating I/O optimization studies.