SecRL-Prune: Structured Reinforcement Learning-Based Pruning of CodeLLMs for Preserving Adversarial Code Mutation

2026-06-04

AI Summary

This study investigates whether structurally pruned large code language models (CodeLLMs) can still generate functionally preserving adversarial code variants—a critical capability for both security evasion and efficient deployment. To this end, the authors propose SecRL-Prune, a novel framework that, for the first time, applies reinforcement learning to channel pruning in CodeLLM feed-forward layers. The approach leverages KL divergence as a reward signal to guide inter-layer pruning strategies and integrates teacher-student distillation with a top-P prediction caching mechanism to substantially reduce GPU memory consumption. Experimental results demonstrate that, at compression rates of 10–30%, three 7B CodeLLMs pruned with SecRL-Prune consistently outperform existing methods in both pass@k and var@k metrics. Notably, variants generated by the 20% pruned models effectively evade malware detection systems.

Abstract

Large code language models (CodeLLMs) can generate and rewrite programs, enabling functionality-preserving code mutation that may be used to create diverse malware variants and evade signature-based detection. A key security question is whether this mutation capability survives model compression, which would make deployment feasible under limited hardware budgets. We propose SecRL-Prune, a structured pruning framework for CodeLLMs that operates on feed-forward (MLP/FFN) channels. Starting from a pretrained teacher, it learns a layer-wise pruning policy with reinforcement learning using a teacher-student KL-divergence reward. To improve efficiency, we cache the teacher's top-P predictions once and compare the pruned student against this compact target, avoiding simultaneous teacher-student residency in GPU memory. We evaluate SecRL-Prune on HumanEval using pass@k for execution correctness and var@k for code diversity across three 7B CodeLLMs at 10-30% compression. SecRL-Prune consistently preserves higher pass@k and var@k than recent structured pruning baselines under aggressive pruning. In a case study on real malware samples, semantics-preserving mutations from 20%-pruned models substantially reduced detections. These results show that code mutation capability can survive significant structured pruning, highlighting the security relevance of compressed CodeLLMs.

Problem

• CodeLLMs
• model compression
• adversarial code mutation
• security
• structured pruning

Innovation

• structured pruning
• reinforcement learning
• CodeLLMs
• adversarial code mutation
• teacher-student distillation