LLM-based Property-based Test Generation for Guardrailing Cyber-Physical Systems

2025-05-29

AI Summary

CPS face challenges including high heterogeneity, difficulty in formalizing security properties, and a disconnect between design-time verification and runtime validation. To address these, this paper proposes an LLM-driven, property-based testing (PBT) methodology for end-to-end assurance. Our approach jointly analyzes source code and documentation to automatically extract formally verifiable system properties and generate highly relevant, executable PBT test cases—requiring minimal human intervention for deployment. Crucially, we co-model property extraction and PBT generation, enabling, for the first time, seamless integration of design-time formal verification with runtime dynamic monitoring. Experimental evaluation demonstrates that our generated PBTs significantly outperform baseline methods in three key dimensions: property relevance, executability, and input-space partition coverage. These results validate the effectiveness and feasibility of leveraging LLMs to establish proactive, adaptive safety guardianship for CPS.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Calibration & Uncertainty Quantification
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior

Application Category

• Security and Privacy: Large-scale security measurements
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Cyber-physical systems (CPSs) are complex systems that integrate physical, computational, and communication subsystems. The heterogeneous nature of these systems makes their safety assurance challenging. In this paper, we propose a novel automated approach for guardrailing cyber-physical systems using property-based tests (PBTs) generated by Large Language Models (LLMs). Our approach employs an LLM to extract properties from the code and documentation of CPSs. Next, we use the LLM to generate PBTs that verify the extracted properties on the CPS. The generated PBTs have two uses. First, they are used to test the CPS before it is deployed, i.e., at design time. Secondly, these PBTs can be used after deployment, i.e., at run time, to monitor the behavior of the system and guardrail it against unsafe states. We implement our approach in ChekProp and conduct preliminary experiments to evaluate the generated PBTs in terms of their relevance (how well they match manually crafted properties), executability (how many run with minimal manual modification), and effectiveness (coverage of the input space partitions). The results of our experiments and evaluation demonstrate a promising path forward for creating guardrails for CPSs using LLM-generated property-based tests.

Problem

• Automated generation of property-based tests for CPS safety
• Using LLMs to extract and verify CPS properties
• Guardrailing CPS at design time and runtime

Innovation

• LLM extracts properties from CPS code
• LLM generates property-based tests automatically
• Tests used pre-deployment and runtime monitoring