Engineering Trustworthy Agentic AI for Critical Systems

2026-07-20

AI Summary

This work addresses the challenge of ensuring trustworthiness in the deployment of autonomous agents within critical engineering systems. It establishes trustworthiness as a core engineering attribute and proposes a unified assurance framework spanning the entire lifecycle—from perception to audit—structured around five key dimensions: safety constraints, robustness, transparency, accountability, and privacy preservation. The study innovatively formulates trustworthiness as a cross-domain commonality and introduces a reusable assurance paradigm analogous to the tiered certification approaches used in safety-critical systems. A systematic technical pathway is developed through integration of multidimensional trust models, architectural analysis, threat modeling, trust mechanisms, and quantitative evaluation metrics. The framework’s cross-domain applicability and effectiveness are validated through common design patterns and failure mode analyses in four representative domains: power systems, autonomous driving, high-performance computing, and communication networks.

Abstract

Agentic artificial intelligence systems, capable of autonomous perception, planning, tool use, and multi-step action, are increasingly proposed for critical engineering domains where decisions carry physical, operational, or economic consequences. This survey addresses a gap in current literature by treating trustworthiness, whether agentic behavior can be verified, audited, and trusted under the constraints that engineering practice actually requires, as a first-class engineering property, rather than evaluating agentic AI by task capability alone. The study adopts a trustworthiness model organized around five cross-cutting dimensions: safety and constraint satisfaction; robustness and reliability; transparency and interpretability; accountability and auditability; and privacy and security. This is mapped onto an agentic assurance workflow spanning perception through audit. Building on this foundation, agentic systems architectures, threats, concrete trust mechanisms, and quantitative metrics are surveyed for direct application in agentic systems development and evaluation. These principles are then examined across four constraint-bound engineering domains: power systems, autonomous vehicles/robotics/UAVs, high-performance computing, and communication networks, identifying recurring design patterns, shared failure modes, and domain-specific gaps. Synthesizing across those domains, agentic AI trustworthiness is shown to be a single problem, with a path outlined toward a reusable, cross-domain assurance framework analogous to the graded certification regimes used by mature safety-critical engineering fields.

Problem

• Agentic AI
• Trustworthiness
• Critical Systems
• Engineering Assurance
• Autonomous Systems

Innovation

• Trustworthy Agentic AI
• Assurance Framework
• Cross-domain Certification
• Safety-critical Systems
• Agentic Workflow