Distilling Knowledge from Large Language Models into Lightweight Reinforcement Learning Agents for Autonomous Cyber Operations

2026-07-30

AI Summary

This study addresses the challenges faced by reinforcement learning (RL) agents in early-stage autonomous cyber defense, including high exploration costs, weak decision-making capabilities, and behavioral instability. To overcome these limitations, the work proposes an online policy distillation framework that leverages a prompt-engineered large language model (LLM) specialized in cybersecurity as a teacher policy. The framework efficiently transfers knowledge from the LLM to a lightweight RL agent containing only 64,910 parameters. Evaluated in multi-scale CybORG network environments with 4 to 12 hosts, the distilled agent closely replicates the teacher’s performance and significantly outperforms baseline RL methods. Despite a five-order-of-magnitude reduction in parameter count, the agent maintains robust defensive capabilities, demonstrating the feasibility of deploying state-of-the-art security models in resource-constrained settings.

Abstract

Autonomous Cyber Operations (ACO) are increasingly important for defending enterprise networks as cyber threats continue to evolve in sophistication. ACO applications commonly employ Reinforcement Learning (RL) agents to learn defensive behaviors through interaction with environments. However, RL agents typically require extensive exploration during training, often resulting in unstable behavior and poor initial decision-making before converging toward effective defense strategies. In this work, we investigate the use of a Large Language Model (LLM) to improve autonomous defensive decision-making within an ACO environment. Through prompt engineering rather than fine-tuning, we demonstrate that an 8-billion parameter LLM pretrained on cybersecurity data can outperform a baseline RL agent in a modified CybORG CAGE Challenge 2 environment. We then propose an online policy distillation framework that transfers the LLM's defensive policy into a lightweight RL agent containing only 64,910 parameters, reducing model size by several orders of magnitude while maintaining effective defensive capabilities. This provides a pathway toward operationalizing frontier cybersecurity models within lightweight, deployable agents. To evaluate transferability, we construct CybORG scenarios ranging from 4 to 12 hosts and assess the approach across varying network configurations. We also evaluate teacher-guided RL stabilization strategies and observe that none consistently surpass the optimized teacher policy, suggesting policy-alignment limitations between reward-driven RL optimization and teacher-guided defense strategies. Our results demonstrate the potential of cybersecurity-focused LLMs as sources of expertise for autonomous cyber defense, while policy distillation provides a practical path toward operationalizing frontier cybersecurity models within efficient, scalable agents.

Problem

• Autonomous Cyber Operations
• Reinforcement Learning
• Policy Distillation
• Large Language Models
• Cyber Defense

Innovation

• policy distillation
• large language models
• autonomous cyber operations
• lightweight reinforcement learning
• prompt engineering