🤖 AI Summary
This work addresses the misalignment between confidence signals from machine learning–based detection models in Security Operations Centers (SOCs) and analysts’ decision-making objectives, which often leads to overlooked false positive costs, increased false negative risks, and alert overload. To bridge this gap, the authors propose a decision-aware trust signal alignment framework that, for the first time, incorporates asymmetric decision costs into trust signal design. By leveraging posterior calibration, lightweight uncertainty indicators, and cost-sensitive thresholds, the approach achieves model-agnostic decision alignment without modifying the underlying detection model. Experiments on the UNSW-NB15 dataset demonstrate that the method reduces weighted loss by several orders of magnitude, substantially decreases false negatives, and lays the groundwork for future human-in-the-loop cybersecurity research.
📝 Abstract
Detection systems that utilize machine learning are progressively implemented at Security Operations Centers (SOCs) to help an analyst to filter through high volumes of security alerts. Practically, such systems tend to reveal probabilistic results or confidence scores which are ill-calibrated and hard to read when under pressure. Qualitative and survey based studies of SOC practice done before reveal that poor alert quality and alert overload greatly augment the burden on the analyst, especially when tool outputs are not coherent with decision requirements, or signal noise. One of the most significant limitations is that model confidence is usually shown without expressing that there are asymmetric costs in decision making where false alarms are much less harmful than missed attacks. The present paper presents a decision-sensitive trust signal correspondence scheme of SOC alert triage. The framework combines confidence that has been calibrated, lightweight uncertainty cues, and cost-sensitive decision thresholds into coherent decision-support layer, instead of making changes to detection models. To enhance probabilistic consistency, the calibration is done using the known post-hoc methods and the uncertainty cues give conservative protection in situations where model certainty is low. To measure the model-independent performance of the suggested model, we apply the Logistic Regression and the Random Forest classifiers to the UNSW-NB15 intrusion detection benchmark. According to simulation findings, false negatives are greatly amplified by the presence of misaligned displays of confidence, whereas cost weighted loss decreases by orders of magnitude between models with decision aligned trust signals. Lastly, we describe a human-in-the-loop study plan that would allow empirically assessing the decision-making of the analysts with aligned and misaligned trust interfaces.