Reliable Remediation Impact Prediction for Black-Box Security Ratings

2026-07-17

AI Summary

This work proposes an applicability-aware surrogate model for black-box security scoring engines, enabling accurate prediction of how remediation actions affect an organization’s security score without revealing the engine’s internal logic. The approach explicitly models the applicability of individual security checks and integrates sensitivity analysis with a reliability assessment layer to identify scenarios where predictions may be unstable. Evaluated on a real-world dataset comprising 5,188 organizational configurations, the proposed model significantly outperforms baseline methods relying on simplistic feature representations. It not only enhances the accuracy of security score predictions but also effectively flags remediation impact estimates that warrant cautious interpretation due to potential unreliability.

Abstract

Security rating platforms summarize externally observable cyber exposure and are expected to help organizations prioritize remediation. A platform may want to tell an organization how a candidate remediation action would affect its score, but repeatedly exposing exact score responses can reveal information about the hidden scoring engine. We propose a surrogate based approach for remediation score impact prediction that is designed to respect this opacity constraint. The surrogate predicts scores from organization configurations while explicitly representing checkpoint (i.e., a security check) applicability and the observed checkpoint set. A main challenge is that such predictions are not uniformly reliable: they depend on the amount and structure of the observable checkpoint evidence available for a given configuration. To address this, the approach combines applicability-aware surrogate construction, sensitivity analysis under controlled checkpoint restriction, a reliability layer for identifying unstable predictions, and score-impact prediction for supported remediation actions. Explicit modeling of checkpoint applicability is central throughout: it improves score prediction and provides the feature basis used by the reliability layer to identify unstable cases. We evaluate the approach on a real-world dataset of 5,188 organization configurations from a commercial security rating platform. The results show that the applicability-aware surrogate improves score prediction over simpler feature representations. For remediation, the surrogate predicts the score impact of supported actions, while the reliability layer helps identify cases in which these predicted impacts should be interpreted cautiously.

Problem

• security ratings
• remediation impact prediction
• black-box systems
• checkpoint applicability
• prediction reliability

Innovation

• surrogate model
• checkpoint applicability
• reliability assessment
• remediation impact prediction
• black-box security ratings