Just TestIt! An SBST Approach To Automate System-Integration Testing

2025-04-10

AI Summary

To address the challenges of automating full-system integration testing, the limited coverage of hardware-software co-faults by conventional formal verification, and the high cost of system performance characterization, this paper proposes TestIt—the first lightweight Software-Based Self-Testing (SBST) integration testing framework tailored for RTL-level systems. TestIt supports dual execution environments: cycle-accurate simulation and FPGA deployment (on PYNQ-Z2), dynamically generating test vectors and golden reference outputs to enable efficient CI/CD integration in open-source RTL development. Innovatively extending SBST to RTL system verification, TestIt detects hardware-software interaction bugs often missed by formal methods and enables low-overhead, rapid system performance profiling. Evaluated on the X-HEEP RISC-V microcontroller, TestIt achieves an 11× speedup on FPGA over RTL simulation and successfully uncovers multiple co-faults undetected by formal verification.

Technology Category

• Machine Learning: Hardware-aware ML
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements

Abstract

This paper introduces TestIt, an open-source Python package designed to automate full-system integration testing using a Software-Based Self-Test (SBST) approach. By dynamically generating test vectors and golden references, TestIt significantly reduces development time and complexity while supporting both simulation and FPGA environments. Its flexible design positions TestIt as a key enabler for the widespread adoption of CI/CD methodologies in open-source RTL development. A case study on the X-HEEP RISC-V microcontroller (MCU), which integrates a custom accelerator, showcases TestIt's ability to detect hardware and software faults that traditional formal methods may overlook. Furthermore, the case study highlights how TestIt can be leveraged to characterize system performance with minimal effort. By automating testing on the PYNQ-Z2 FPGA development board, we achieved a 11x speed-up with respect to RTL simulations.

Problem

• Automate full-system integration testing using SBST
• Detect hardware and software faults overlooked by formal methods
• Characterize system performance with minimal effort

Innovation

• Automates system testing with SBST approach
• Generates test vectors dynamically for efficiency
• Supports both simulation and FPGA environments