model validation

This study addresses the frequent violations of clinical coding standards—such as ICD-10, CPT, and HL7 FHIR—by large language models when generating structured medical data, which impedes integration with electronic health record systems. To mitigate this, the authors propose and validate a closed-loop verification-and-repair framework that automatically detects and iteratively corrects formatting errors. The approach is evaluated using three open-source models—Qwen2.5-7B, Llama3.1-8B, and Gemma2-9B—deployed locally across 320 clinical scenarios. Results demonstrate a substantial improvement in schema compliance across all models, achieving an overall adherence rate of 99.0% and increasing individual model performance by 7.8 to 12.5 percentage points. Notably, 96% of detected errors were attributable to repairable representation-layer issues, with most resolved within one or two correction rounds, effectively compensating for the models’ limited understanding of healthcare IT standards.

This work addresses the challenge of providing verifiable safety assurance for robotic systems in safety-critical domains, where traditional assurance cases rely on manually generated evidence that is costly, error-prone, and difficult to maintain. The paper proposes a model-based automated approach that deeply integrates formal verification into the assurance workflow. It employs RoboChart—a domain-specific modeling language with formal semantics—to capture system designs, and introduces a template-driven mechanism to automatically translate natural-language requirements into formal assertions. These assertions are then discharged through a combination of model checking and theorem proving tools, yielding formally verified evidence that can be seamlessly integrated into assurance cases. Case studies demonstrate that the proposed method significantly enhances the reliability, maintainability, and degree of automation in safety argumentation.

Current verification workflows for autonomous systems suffer from a lack of coordination among scenario design, simulation execution, and telemetry analysis, leading to poor traceability between requirements, tests, and evidence, which undermines reproducibility and debugging efficiency. This work proposes a unified verification framework powered by large language models (LLMs) that bridges this gap through task-level structured scenario representations. The framework automatically translates high-level verification intents into temporally evolving scenarios, enabling automated simulation execution and context-aligned telemetry analysis. Furthermore, it incorporates a counterfactual scenario generation mechanism driven by failure cases to establish a closed-loop, self-evolving testing process. The approach substantially enhances traceability, reproducibility, and scalability of verification, accelerates test iteration cycles, and deepens insight into system behavior.

This study addresses the limitations of existing SysML verification approaches, which are often tool-dependent and restricted to performance properties, lacking support for automated validation of behavioral and interface requirements. To overcome these shortcomings, this work proposes a tool-agnostic, automated verification workflow driven by SysML test cases, integrating UML Testing Profile and behavioral diagram constructs to enable unified validation of multidimensional attributes—including behavior, timing, and state responses. The methodology was developed through a mixed-methods research strategy combining literature review and stakeholder interviews, and its efficacy was empirically validated across two independent SysML toolchains. The approach not only transcends the constraints of conventional parametric methods but also enables automatic traceability of verification results back to the original model elements.

This work addresses the inefficiency and high cost of compliance testing in highly regulated domains, where current practices rely on manual translation of regulations into test cases by experts. While large language models (LLMs) offer automation potential, they often suffer from hallucination, and existing hybrid approaches still require significant human modeling effort. To overcome these limitations, the authors propose RAFT, a novel framework that explicitly extracts implicit regulatory knowledge from multiple LLMs and leverages an adaptive purification-aggregation strategy with dynamic prompt injection to automatically generate domain-specific meta-models, formalized requirements, and testability constraints—enabling fully automated, human-intervention-free compliance test generation. Experiments in financial, automotive, and power sectors demonstrate that RAFT achieves expert-level performance, significantly outperforming state-of-the-art methods while drastically reducing test case generation and review time.

This study addresses the lack of traceable, structured linkage between high-level requirements and low-level automated testing in AI-enabled cyber-physical systems, which hinders compliance with regulatory demands for verifiable evidence. To bridge this gap, the paper introduces VNVSpec, a novel framework that enables end-to-end automated traceability and closed-loop verification from high-level engineering requirements to test cases. VNVSpec employs machine-readable verification and validation (V&V) specifications to support requirement ingestion, quality checks, metric-driven decomposition, test result association, and generation of audit-ready reports, all integrated into a continuous integration pipeline. Empirical evaluation demonstrates that the approach verifies 36 requirements against 449 tests in linear time, scales to tens of thousands of artifacts, and is fully reproducible through open-sourced code, test suites, and benchmark scripts.

This work addresses the challenge of reliably translating natural language into industrial-grade, deployable SysMLv2 models. The authors propose an iterative generate-check-repair framework that, for the first time, integrates a production-level SysMLv2 conformance checker directly into the generation process as a control mechanism rather than a post-processing step. By combining large language model (LLM) generation with deterministic diagnostic feedback and targeted repair strategies—and terminating only when zero errors remain—the method achieves perfect compliance. Evaluated across 604 test cases derived from 151 prompts and four distinct LLMs, the approach elevates single-pass generation compliance from 51.16% to 100%, enabling robust, direct translation of natural language specifications into engineering-ready SysMLv2 models.

This work addresses the lack of standardized governance in AI-assisted development of scientific software, which currently hinders compliance with stringent quality assurance requirements such as ASME NQA-1. The authors propose a structured framework that, for the first time, integrates large language models with software verification and validation (V&V) methodologies under NQA-1 compliance. Using the open-source nuclear-grade code TMAP8 as a testbed, they establish an AI-assisted V&V use case development process that is verifiable, traceable, and auditable. The framework explicitly defines mechanisms for disclosing AI-generated content, subjecting it to rigorous review, and assigning human accountability, thereby unifying transparency, reproducibility, and regulatory compliance. This approach ensures software correctness while meeting the rigorous demands of high-assurance quality standards.

This study addresses the underexplored engineering challenges in existing machine learning evaluation frameworks, where operational issues and their root causes have lacked systematic investigation. To bridge this gap, the work formally establishes evaluation engineering as a distinct research direction within software engineering. Through an empirical analysis of 57 frameworks and a comprehensive categorization of 16,560 reported issues across a newly proposed five-stage workflow model, the study reveals that 41.4% of problems originate in the specification phase, while 61.7% of classified issues stem from missing functionality, inadequate documentation, and insufficient input validation. The findings yield a structured taxonomy of evaluation-related problems and provide empirical evidence to inform the design and improvement of robust evaluation systems.

This work addresses the challenge that domain experts face in translating natural language descriptions of data quality requirements into executable analyses, a process often hindered by reliance on data engineers, resulting in inefficiency and high technical barriers. To overcome this, the paper proposes a no-code, model-driven pipeline that leverages a QPM metamodel to define domain-specific quality analysis templates. Coupled with the Constrainify toolchain, it automatically transforms natural language requirements into executable and reusable analytical logic. By integrating model-driven engineering, metamodeling, and no-code web technologies, the approach significantly reduces dependency on technical expertise, enabling efficient, reproducible, and semantically aligned data quality assessments. This advancement enhances both the accessibility and automation of data quality analysis for non-technical domain practitioners.