Comparative Model Fidelity Evaluation to Support Design Decisions for Complex, Novel Systems of Systems

2025-08-04

AI Summary

In the pre-prototype phase of complex novel systems, the absence of empirical data impedes rigorous assessment of simulation model credibility. Method: This paper proposes a physics-fidelity-based model trust evaluation method that bypasses reliance on real-world measurements. Instead, it quantifies model applicability by systematically analyzing the completeness of represented physical phenomena, the mathematical complexity of their formulation, and the fidelity of emergent behavior modeling. Contribution/Results: The approach enables objective, quantitative ranking of multiple candidate models under data-scarce conditions—thereby significantly enhancing the reliability of simulation-driven decisions during early-stage design. It establishes both theoretical foundations and practical tools for model-based design in high-uncertainty scenarios, advancing trustworthy digital twin development and physics-informed simulation validation.

Technology Category

• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Computer Vision: Low Level & Physics-based Vision
• Data Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Web Mining and Content Analysis: Web data generation and simulation
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Systems design processes are increasingly reliant on simulation models to inform design decisions. A pervasive issue within the systems engineering community is trusting in the models used to make decisions about complex systems. This work presents a method of evaluating the trustworthiness of a model to provide utility to a designer making a decision within a design process. Trusting the results of a model is especially important in design processes where the system is complex, novel, or displays emergent phenomena. Additionally, systems that are in the pre-prototype stages of development often do not have sources of ground truth for validating the models. Developing methods of model validation and trust that do not require real-world data is a key challenge facing systems engineers. Model fidelity in this work refers to the adherence of a model to real-world physics and is closely tied to model trust and model validity. Trust and validity directly support a designer's ability to make decisions using physics-based models. The physics that are captured in a model and the complexity of the mathematical representation of the physics contribute to a model's fidelity, and this work leverages the included physical phenomena to develop a means of selecting the most appropriate for a given design decision.

Problem

• Evaluating trustworthiness of simulation models for complex systems
• Validating models without real-world data in pre-prototype stages
• Selecting appropriate physics-based models for design decisions

Innovation

• Evaluates model trustworthiness without real-world data
• Leverages physical phenomena for model fidelity
• Supports design decisions with physics-based models