SimuVerity: Benchmarking Agents for Engineering-Grade Simulink Model Generation

2026-10-01

AI Summary

This study addresses the limitation of existing generative model benchmarks, which focus solely on code compilation or structural similarity and fail to verify engineering compliance. We construct a text-to-executable Simulink model generation benchmark spanning ten domains and propose an executable system archive-based method for aligning with engineering specifications. Furthermore, we design a hierarchical automated evaluation mechanism alongside a six-dimensional dynamic response metric system, enabling end-to-end assessment from deliverability and executability to engineering qualification. Experimental results demonstrate that the best-performing agent achieves a score of only 42.86, confirming a significant disconnect between structural similarity and engineering performance. These findings reveal critical capability bottlenecks in current large language models regarding the generation of qualified systems that satisfy multidimensional engineering requirements.

Abstract

Existing Simulink benchmarks mainly evaluate whether generated models compile, execute, or resemble a reference model. These criteria do not establish whether a model satisfies its engineering requirements. We introduce SimuVerity, a benchmark of 101 text-to-executable Simulink model-generation tasks across ten engineering domains. For each task, executable-system profiles ground the engineering specification and four families of native simulation scenarios. A hierarchical evaluator first checks artifact delivery, native executability, and engineering qualification, then scores qualified models across six dimensions covering accuracy, output quality, mechanistic fidelity, control and causal integrity, operating-domain robustness, and dynamic response. We evaluate six agent systems with SimuVerity. The best system achieves an overall score of only 42.86. The results show that structural similarity is a poor proxy for engineering performance: capability bottlenecks arise both in producing qualified implementations and in satisfying multidimensional requirements after qualification. Meanwhile, some high-scoring models still exhibit severe visual-layout disorder. SimuVerity provides a systematic basis for assessing agents' engineering capabilities and diagnosing failures in executable Simulink model generation.

Problem

• Simulink model generation
• benchmarking agents
• engineering requirements
• text-to-Simulink
• model evaluation

Innovation

• Simulink model generation
• benchmarking
• hierarchical evaluation
• engineering requirements
• LLM agents