Small, Free, and Effective: Orchestrating Open-Weight Small Language Models to Outperform Single LLM for Malware Analysis

2026-07-22

AI Summary

This work addresses the challenges of applying large language models to malware analysis in resource-constrained settings, where closed-source models incur high costs and lack transparency, while individual open-source small models exhibit limited performance. To overcome these limitations, the authors propose a multi-agent collaborative framework that integrates four coordination strategies—including an evidence-driven pipeline and an adversarial debate mechanism—to orchestrate open-source models such as Qwen3-4B and Foundation-Sec-8B in jointly interpreting file system, network, and process behaviors from malware detonation reports. Evaluated on the CyberSecEval benchmark, the proposed hybrid architecture achieves an accuracy of 35.30%, significantly outperforming both the strongest specialized baseline (22.54%) and state-of-the-art large models without evidence guidance (34.77%), thereby demonstrating the efficacy and novelty of collaborative reasoning in security analysis.

Abstract

Malware analysis demands rapid interpretation of complex detonation reports spanning filesystem, network, and process behaviours. While large language models (LLMs) demonstrate impressive capabilities for technical artifact interpretation, the opacity and escalating API costs of closed-weight frontier models motivate exploration of open-weight alternatives. However, many open-weight models are large, demanding significant compute resources and incurring non-trivial hosting costs that place them beyond reach for resource-constrained deployments. This paper investigates whether orchestrated ensembles of small language models (SLMs) can match or exceed single LLM performance on structured questions about malware detonation reports. We established baselines by testing eleven open-weight SLMs, three cyber security pre-trained models, and six frontier LLMs on Meta's CyberSecEval Malware Analysis benchmark. We then designed and evaluated four orchestration architectures: (i) a multi-agent pipeline that decomposes analysis into structured evidence-collection and reasoning stages, (ii) an adversarial debate framework in which two agents iteratively critique each other's reasoning, (iii) a hierarchical consultation system that pairs a general-purpose SLM with a cyber-specialised expert model, and (iv) a hybrid architecture that combines evidence-grounded pipelines with adversarial debate reasoning. The hybrid system (Qwen3-4B with Foundation-Sec-8B) achieved 35.30% overall accuracy, exceeding the strongest cyber-specialised baseline (22.54%) and the strongest ungrounded frontier baseline (34.77%); when given the same evidence pipeline, grounded Gemini remained the strongest configuration at 38.22%. These findings show that evidence-grounded orchestration can substantially improve the performance of collaborative SLMs for supporting interpretation of malware detonation reports.

Problem

• malware analysis
• small language models
• open-weight models
• detonation reports
• resource-constrained deployment

Innovation

• small language models
• model orchestration
• malware analysis
• evidence-grounded reasoning
• adversarial debate