The Competence Shadow: Theory and Bounds of AI Assistance in Safety Engineering

2026-03-26

AI Summary

This study addresses how AI-assisted safety engineering may systematically narrow human reasoning due to “capability shadows”—implicit omissions of critical information by AI—leading to safety flaws that only emerge post-deployment. To tackle this, the work proposes a five-dimensional capability framework that formalizes human–AI collaboration structures, reframing the challenge as one of collaborative workflow design. Through formal modeling and boundary analysis, the paper reveals that capability shadows exhibit multiplicative accumulation effects—far exceeding linear estimates—and derives closed-form bounds on performance degradation alongside conditions for non-degradation. The research advocates shifting certification paradigms from individual AI tools to entire collaborative workflows, thereby establishing a theoretical foundation and design principles for safe human–AI interaction in safety-critical domains.

Technology Category

• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Humans and AI: Human-AI Collaboration / Human-AI Teaming
• Natural Language Processing: Safety and Robustness

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

As AI assistants become integrated into safety engineering workflows for Physical AI systems, a critical question emerges: does AI assistance improve safety analysis quality, or introduce systematic blind spots that surface only through post-deployment incidents? This paper develops a formal framework for AI assistance in safety analysis. We first establish why safety engineering resists benchmark-driven evaluation: safety competence is irreducibly multidimensional, constrained by context-dependent correctness, inherent incompleteness, and legitimate expert disagreement. We formalize this through a five-dimensional competence framework capturing domain knowledge, standards expertise, operational experience, contextual understanding, and judgment. We introduce the competence shadow: the systematic narrowing of human reasoning induced by AI-generated safety analysis. The shadow is not what the AI presents, but what it prevents from being considered. We formalize four canonical human-AI collaboration structures and derive closed-form performance bounds, demonstrating that the competence shadow compounds multiplicatively to produce degradation far exceeding naive additive estimates. The central finding is that AI assistance in safety engineering is a collaboration design problem, not a software procurement decision. The same tool degrades or improves analysis quality depending entirely on how it is used. We derive non-degradation conditions for shadow-resistant workflows and call for a shift from tool qualification toward workflow qualification for trustworthy Physical AI.

Problem

• AI assistance
• safety engineering
• competence shadow
• Physical AI
• human-AI collaboration

Innovation

• competence shadow
• safety engineering
• human-AI collaboration
• workflow qualification
• Physical AI