Pareto-optimal quantum kernel selection for unsupervised anomaly detection on real malware beaconing data

2026-10-07

AI Summary

This study addresses the challenge of jointly evaluating detection performance and quantum geometric advantage in quantum kernel anomaly detection by proposing a pioneering unsupervised multi-objective Pareto optimization framework. The method simultaneously optimizes pseudo-discrepancy and geometric difference metrics, selecting optimal quantum kernel models from the Pareto front to effectively balance detection accuracy with quantum advantage verification. Experiments employ fidelity and projected quantum kernels with one-class support vector machines, validated on both the IQM Garnet processor and simulators. Results demonstrate that the optimal model achieves an AUC of 0.840, significantly outperforming classical baselines, while elucidating the quantum advantage boundaries associated with high-geometric-difference models.

Abstract

Quantum kernel methods are leading candidates for a practical quantum advantage in machine learning, but assessing that potential requires two quantities usually reported separately: how well a kernel performs on the task, and how far its geometry departs from the classical kernels available for the same problem. We introduce a fully unsupervised, multi-objective protocol that optimises simultaneously the normalised pseudo discrepancy (NPD), a label-free proxy for anomaly detection quality, and the geometric difference (GD) to a tuned classical reference kernel, selecting models from the resulting Pareto front. We apply it to malware beaconing detection in real network traffic, using a one-class support vector machine with fidelity and projected quantum kernels over four data encodings, on simulators and on IQM's 20-qubit Garnet processor. NPD-guided selection alone finds a fidelity kernel that beats the tuned classical baseline, but with a geometric difference too small to certify the gain as quantum. Projected kernels reach far larger geometric differences; the Pareto-selected one only marginally exceeds the baseline (AUC $0.782$ versus $0.765$, $g_{C\to Q}\approx 89>\sqrt{N}$ relative to that reference kernel), still below the NPD-selected fidelity kernel ($0.840$).

Problem

• quantum kernel selection
• unsupervised anomaly detection
• malware beaconing
• Pareto optimality
• geometric difference

Innovation

• Quantum kernel selection
• Pareto-optimal optimization
• Unsupervised anomaly detection
• Geometric difference
• Malware beaconing detection