🤖 AI Summary
Evaluating large language models’ (LLMs) capability to generate accurate causal loop diagrams (CLDs) for system dynamics (SD) modeling remains an open challenge due to the lack of standardized, domain-specific benchmarks.
Method: This work introduces the first dual-dimensional SD-oriented evaluation framework, assessing both *technical correctness* (accuracy of causal translation from textual descriptions to CLDs) and *instruction fidelity* (adherence to structured prompting requirements). We develop the open-source sd-ai evaluation engine and a standardized test suite, integrating multi-LLM causal reasoning assessment (e.g., GPT-4.5-preview, o1, GPT-4o), structured prompt engineering, and human-in-the-loop validation.
Contribution/Results: Empirical evaluation across 11 state-of-the-art LLMs reveals substantial performance variance—e.g., o1 achieves 100% causal translation accuracy, while GPT-4.5-preview attains a composite score of 92.9%. The framework demonstrates high reproducibility and community extensibility, establishing the first standardized benchmark for AI-driven SD modeling and enabling rigorous, comparable assessment of dynamic systems reasoning capabilities.
📝 Abstract
Introduction: As system dynamics (SD) embraces automation, AI offers efficiency but risks bias from missing data and flawed models. Models that omit multiple perspectives and data threaten model quality, whether created by humans or with the assistance of AI. To reduce uncertainty about how well AI can build SD models, we introduce two metrics for evaluation of AI-generated causal maps: technical correctness (causal translation) and adherence to instructions (conformance). Approach: We developed an open source project called sd-ai to provide a basis for collaboration in the SD community, aiming to fully harness the potential of AI based tools like ChatGPT for dynamic modeling. Additionally, we created an evaluation theory along with a comprehensive suite of tests designed to evaluate any such tools developed within the sd-ai ecosystem. Results: We tested 11 different LLMs on their ability to do causal translation as well as conform to user instruction. gpt-4.5-preview was the top performer, scoring 92.9% overall, excelling in both tasks. o1 scored 100% in causal translation. gpt-4o identified all causal links but struggled with positive polarity in decreasing terms. While gpt-4.5-preview and o1 are most accurate, gpt-4o is the cheapest. Discussion: Causal translation and conformance tests applied to the sd-ai engine reveal significant variations across lLLMs, underscoring the need for continued evaluation to ensure responsible development of AI tools for dynamic modeling. To address this, an open collaboration among tool developers, modelers, and stakeholders is launched to standardize measures for evaluating the capacity of AI tools to improve the modeling process.