ORCAGen: Orchestrating Context-Aware Malware Deception with RAG-Guided Generative AI

2026-10-08

AI Summary

Traditional malware defense relies on rapid isolation, forfeiting opportunities for behavioral observation and proactive countermeasures. This work proposes an active deception framework leveraging Retrieval-Augmented Generation (RAG) and knowledge graphs. Specifically, structured prompt engineering guides large language models to generate executable deception strategies. Following proof-of-concept pre-testing to eliminate hallucinations, offline playbooks are constructed to enforce deterministic logic at runtime. Multi-model evaluations demonstrate that this approach enables lightweight deployment: GPT-5.5 achieves zero hallucination, while Gemini maintains low computational overhead. Furthermore, the framework effectively disrupts real-world malware samples, including keyloggers, validating its practical utility for proactive cyber defense.

Abstract

Malware defenses often remove or isolate suspicious programs as quickly as possible. While effective for containment, this approach can also waste an opportunity to observe attacker behavior and deploy targeted countermeasures. ORCAGen takes a different approach: it uses GenAI to build malware-specific deception playbooks offline, validates them before deployment, and enforces only the verified logic at runtime. ORCAGen combines Retrieval-Augmented Generation (RAG) with structured prompt engineering to generate both proof-of-concept (PoC) malware and corresponding deception orchestration code. A curated knowledge base (KB) of malware procedures and active defense strategies grounds the generation process, helping the LLM produce threat-specific and executable deception logic rather than generic or hallucinated outputs. The generated PoC malware provides a safe and reproducible way to test whether a deception strategy can disrupt, redirect, or suppress targeted malware behavior before the strategy is added to the runtime playbook. We evaluate ORCAGen across GPT-4o, GPT-5.5, Gemini 3.5 Flash, Qwen3-Coder, and Claude Sonnet 4.5 using execution success, hallucination rate, refinement effort, deception effectiveness, and runtime overhead. The evaluation covers synthesized malware scenarios and 150 real-world malware samples across keyloggers, information stealers, and ransomware. Across the evaluated scenarios, GPT-5.5 required the fewest refinements and produced no observed hallucinated APIs, while Gemini 3.5 Flash achieved the lowest response time and runtime overhead. The results show that RAG-guided structured prompting can support the scalable construction of malware-specific deception playbooks that remain lightweight and deterministic during runtime enforcement.

Problem

• Malware Deception
• Active Defense
• Context-Aware Security
• Threat Observation

Innovation

• Retrieval-Augmented Generation
• Generative AI
• Malware Deception
• Structured Prompt Engineering
• Active Defense