What the Detector Can See: Evaluating CPS Anomaly Detectors Independently of the Decision Rule

2026-08-03

AI Summary

This work addresses the confounding of physical process modeling capability and alarm thresholding effects in existing evaluations of cyber-physical system (CPS) anomaly detectors, which obscures the attribution of performance differences. To resolve this, the authors decouple detection into two stages—residual generation and threshold-based alarming—and propose a normalized residual energy–based evaluation metric. This metric independently quantifies a model’s ability to represent the underlying physical process without relying on specific decision rules or hyperparameter tuning. Furthermore, it connects to KL divergence to measure attack separability, training–testing stability, and model compactness. Evaluations across five detector families on the SWaT, WADI, and HAI benchmarks reveal that performance rankings are highly scenario-dependent and precisely identify failure causes—such as inadequate representation capacity, suboptimal thresholds, or weak physical manifestations of attacks.

Abstract

Anomaly detectors are often the last line of defense for cyber-physical systems (CPS). But detectors built in very different ways, from deep neural networks to invariant templates, are usually compared using precision, recall, or F1 at a single operating point. These scores mix two separate things: how well the detector represents the physical process, and how well its alarm threshold is set. We therefore treat a CPS anomaly detector as a two-stage pipeline: Stage 1 maps observations to residuals, and Stage 2 maps residuals to alarms. Instead of scoring only the final alarms, we evaluate Stage 1 directly using normalized residual energy, which has an exact connection to the Kullback-Leibler divergence from the trained-normal reference distribution. Because it does not depend on a specific alarm rule, it can separately measure attack separation, stability across the train-test gap, and the compactness with which a detector encodes the plant. Without any per-detector tuning, we apply this evaluation to five detectors -- GDN, FuSAGNet, TranAD, NSIBF, and GeCo -- across three CPS benchmarks: SWaT, WADI, and HAI. Although the detectors have similar ROC-AUC values on SWaT, their performance differs by more than an order of magnitude at a common false-alarm rate. Rankings also change across testbeds: TranAD ranks first on HAI but last on SWaT, while NSIBF ranks first on WADI but last on HAI. On WADI, localized attacks can evade detectors that pool evidence across all channels, helping explain why NSIBF outperforms methods that do well on other benchmarks. These results show that detection failure can come from different sources: a weak representation, poor threshold calibration, or an attack with little physical effect. A decision-rule-free analysis helps separate these causes.

Problem

• anomaly detection
• cyber-physical systems
• evaluation methodology
• detector representation
• decision rule

Innovation

• anomaly detection
• cyber-physical systems
• residual evaluation
• decision-rule-free analysis
• normalized residual energy