PairAudit: Guiding Human Review with Graph Tokens under Distribution Shift

2026-10-07

AI Summary

This study addresses the challenge of intrusion detectors producing high-confidence misclassifications on unseen attacks under distribution shift, particularly when manual review budgets are constrained. We propose a retraining-free graph token anomaly relationship mining method that leverages graph neural networks and uncertainty quantification to capture anomalous patterns among nodes, thereby identifying latent errors obscured by existing predictions. Furthermore, this approach incorporates a human-in-the-loop feedback mechanism to dynamically optimize review prioritization. Experimental results demonstrate that, under a fixed review budget, the proposed method corrects significantly more errors than conventional uncertainty-based baselines and substantially improves the detection rate of previously unseen attacks.

Abstract

Intrusion detectors can confidently misclassify attacks that were not seen during training. Human review can correct these errors, but only a limited number of cases can be checked. Uncertainty-based review may overlook confident errors, while anomaly scores alone do not show whether changing the review plan will correct more errors. We introduce PairAudit to find overlooked errors and improve review under a fixed budget. Its graph tokens capture prediction patterns across connected nodes. Rather than building another predictor through feature aggregation, PairAudit uses unusual relational patterns to uncover potential errors in existing predictions. Human feedback then helps decide whether these findings justify changing review priorities. Experiments across security tasks show that PairAudit corrects more errors on average than uncertainty-based review, including more errors on unseen attacks. These gains account for all review costs and do not require retraining the detector.

Problem

• intrusion detection
• distribution shift
• human review
• confident misclassification
• budget constraint

Innovation

• Graph Tokens
• PairAudit
• Distribution Shift
• Human Review
• Intrusion Detection