Adaptive Fault Injection Planning for Multi-Layer Self-Healing AI Infrastructure

2026-07-17

AI Summary

This study addresses the limitations of traditional fault injection methods, which test individual layers in isolation and fail to capture cross-layer fault propagation, leaving critical vulnerabilities undetected in multi-layer self-healing AI infrastructures. To overcome this, the authors propose ADA-ST, a novel approach that introduces cross-layer modeling via weighted fault propagation graphs and an adaptive injection mechanism guided by an activity-aware scenario selection algorithm to maximize marginal coverage gain. The method further incorporates Fault-Layer Abstraction Mapping (FLAM) to enable knowledge transfer of fault propagation patterns across hardware generations. Experimental results demonstrate that ADA-ST achieves 100% edge coverage on Alpha, Beta, and Gamma platforms within only 9–12 iterations, with FLAM achieving 96%–100% transfer accuracy. Physical validation confirms all four predicted propagation paths, uncovering four new classes of cross-layer vulnerabilities and critical blind spots in self-healing mechanisms.

Abstract

Modern GPU-accelerator platforms rely on multi-layer self-healing pipelines that span hardware, firmware, management software, and orchestration. When faults propagate across layer boundaries, they can bypass detection, corrupt diagnosis, or trigger conflicting remediations--yet conventional fault-injection campaigns test each layer in isolation. We present ADA-ST, an adaptive fault-injection methodology that uses a weighted fault-propagation graph to guide cross-layer scenario selection. We construct four-layer graphs for three successive platforms at a hyperscale operator: Alpha, Beta, and Gamma. Platform Alpha, a production system that accumulated 72,550 repair tickets over four years, provides the empirical foundation; 49% of those tickets involve cross-layer fault propagation. We show that existing static test campaigns cover only 20-25% of the modeled fault-propagation edges, leaving approximately three-quarters of the cross-layer attack surface unexercised. ADA-ST closes this gap through iterative, activity-guided scenario selection that maximizes marginal coverage gain per iteration, reaching full edge coverage within 10 iterations on Alpha, 12 on Beta, and 9 on Gamma. The Fault-Layer Abstraction Mapping (FLAM) transfers propagation knowledge across hardware generations with 100% fidelity from Alpha to Beta and 96% from Beta to Gamma. Physical spot-validation on the newest platform confirms all four tested propagation edges, revealing cross-layer vulnerabilities spanning telemetry blind spots, absence-based detection gaps, multi-signal correlation failures, and trust-without-verification propagation at the L2-to-L3 boundary.

Problem

• fault propagation
• cross-layer
• self-healing
• GPU-accelerator platforms
• fault injection

Innovation

• adaptive fault injection
• cross-layer fault propagation
• self-healing infrastructure
• fault-layer abstraction mapping
• coverage-guided testing