🤖 AI Summary
This work addresses the lack of a unified framework for runtime monitoring in safety-critical machine learning systems, which hinders systematic design and evaluation. The authors propose a general runtime monitoring framework that, for the first time, systematically categorizes monitoring mechanisms into three complementary dimensions: Operational Design Domain (ODD) compliance checking, out-of-distribution (OOD) detection, and out-of-model-scope (OMS) anomaly analysis. They further establish a safety-oriented, unified evaluation methodology grounded in this tripartite structure. Demonstrated on a vision-based aircraft runway detection task, the framework significantly enhances the identification of unsafe inputs and anomalous model behaviors, enables fair cross-method comparison and integration, and thereby advances the trustworthy deployment of safety-critical AI systems.
📝 Abstract
Runtime monitoring is essential to ensure the safety of ML applications in safety-critical domains. However, current research is fragmented, with independent methods emerging from different communities. In this paper, we propose a unified framework categorising runtime monitoring approaches into three distinct types: Operational Design Domain (ODD) monitoring, which ensures compliance with expected operating conditions; Out-of-Distribution (OOD) monitoring, which rejects inputs that deviate from the training data; and Out-of-Model-Scope (OMS) monitoring, which detects anomalous model behaviour based its internal states or outputs. We demonstrate the benefits of this categorization with a dedicated experiment on an aeronautical safety-critical application: runway detection during landing. This framework facilitates design of monitoring activities, with complementary categories of monitors, and enables evaluation and comparison of different monitors using common, safety-oriented metrics.