Multi-Objective Reinforcement Learning for Automated Resilient Cyber Defence

2024-11-26

arXiv.org

AI Summary

Autonomous defense in military and critical infrastructure networks faces inherent trade-offs among conflicting objectives—such as attack suppression and business continuity assurance—posing significant challenges for conventional single-objective reinforcement learning (RL). Method: This work pioneers the integration of multi-objective reinforcement learning (MORL) into cyber offense-defense games, proposing two online-adaptive autonomous cyber defense (ACD) agents: Multi-Objective Proximal Policy Optimization (MOPPO) and Pareto-Conditioned Network (PCN). Both incorporate Pareto-optimality constraints into the PPO framework and dynamically adapt to operational constraints within a high-fidelity autonomous cyber defense simulation environment. Contribution/Results: Experimental evaluation demonstrates that the proposed MORL agents achieve Pareto-superior balance across attack interception rate, service availability, and recovery cost—outperforming single-objective RL baselines significantly. This advances beyond the fundamental limitation of traditional RL in simultaneously optimizing multiple, often competing, security operation objectives.

Technology Category

Multiagent Systems: Adversarial AgentsSearch and Optimization: Learning to SearchMachine Learning: Adversarial Learning & Robustness

Application Category

Responsible Web: Machine-in-the-loop, human agency and autonomyEconomics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystemsSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Cyber-attacks pose a security threat to military command and control networks, Intelligence, Surveillance, and Reconnaissance (ISR) systems, and civilian critical national infrastructure. The use of artificial intelligence and autonomous agents in these attacks increases the scale, range, and complexity of this threat and the subsequent disruption they cause. Autonomous Cyber Defence (ACD) agents aim to mitigate this threat by responding at machine speed and at the scale required to address the problem. Sequential decision-making algorithms such as Deep Reinforcement Learning (RL) provide a promising route to create ACD agents. These algorithms focus on a single objective such as minimizing the intrusion of red agents on the network, by using a handcrafted weighted sum of rewards. This approach removes the ability to adapt the model during inference, and fails to address the many competing objectives present when operating and protecting these networks. Conflicting objectives, such as restoring a machine from a back-up image, must be carefully balanced with the cost of associated down-time, or the disruption to network traffic or services that might result. Instead of pursing a Single-Objective RL (SORL) approach, here we present a simple example of a multi-objective network defence game that requires consideration of both defending the network against red-agents and maintaining critical functionality of green-agents. Two Multi-Objective Reinforcement Learning (MORL) algorithms, namely Multi-Objective Proximal Policy Optimization (MOPPO), and Pareto-Conditioned Networks (PCN), are used to create two trained ACD agents whose performance is compared on our Multi-Objective Cyber Defence game. The benefits and limitations of MORL ACD agents in comparison to SORL ACD agents are discussed based on the investigations of this game.

Problem

• Addressing cyber threats to military and civilian networks using autonomous agents
• Balancing competing objectives in cyber defense like security and functionality
• Comparing multi-objective reinforcement learning algorithms for resilient cyber defense

Innovation

• Multi-Objective Reinforcement Learning for cyber defense
• MOPPO and PCN algorithms for autonomous agents
• Balancing network defense and critical functionality