Behavioral Grammar: Detecting Adaptive Malware via Tiny Language Model Priors and Second-Order Temporal Analysis

2026-08-01

AI Summary

Current endpoint detection systems struggle to simultaneously achieve scalability, auditability, and effective detection of adaptive polymorphic malware. This work proposes a behavioral grammar framework that models host runtime behavior as a structured language, leveraging a compact causal Transformer with only 0.88M parameters to learn its syntactic regularities. By integrating second-order temporal analysis, coefficient of variation of event intervals, and prototype learning, the approach enables both anomaly detection and attack attribution. The method uncovers a fundamental trade-off attackers face between stealth and functionality, achieving a 93% detection rate against adaptive adversaries capable of behavioral mimicry, with an initial false positive rate of 3.84%. Notably, the coefficient of variation of event intervals between benign and malicious behaviors differs by a factor of 30 (9.786 vs. 0.310), substantially enhancing defensive asymmetry.

Abstract

Modern endpoint detection systems face a fundamental tension: signature-based approaches are trivially evaded by polymorphic or adaptive threats, while heavy deep-learning models resist auditability and deployment at scale. This paper presents Behavioral Grammar, a detection architecture that treats host runtime behavior as a structured language and learns its grammar with a compact 0.88M-parameter causal Transformer (TinyGPT). Each system event is discretized into an 8-token representation spanning event type, process, argument skeleton, path category, parent process, user, destination, and inter-event timing. The model learns the conditional distribution of normal behavior in a purely self-supervised manner, and anomaly scores are derived from per-slot negative log-likelihood (NLL) statistics, yielding a mathematically bounded false-positive rate. We augment this prior with prototype learning for known-attack attribution, second-order temporal analysis for cadence-based detection, self-learning pattern extraction, and a five-network fusion pipeline. Against an Adaptive Adversarial Agent (AAA)--a threat that learns survival strategies under defensive pressure, performs behavioral mimicry, and matches host event rates--our system achieves 93% detection at 3.84% onboarding false-positive rate. The strongest discriminative signal arises not from any single event but from the coefficient of variation of inter-event intervals: the AAA stepping cadence exhibits CV=0.310 versus 9.786 for benign sleep intervals, a 30x separation reflecting a fundamental stealth-functionality trade-off. We frame these findings within a coevolutionary economics model, arguing that behavior-grammar detection shifts the evasion cost from rule circumvention (cheap) to distribution matching (expensive), establishing a structural asymmetry favoring the defender.

Problem

• adaptive malware
• endpoint detection
• behavioral analysis
• evasion resistance
• anomaly detection

Innovation

• Behavioral Grammar
• Tiny Language Model
• Second-Order Temporal Analysis
• Adaptive Malware Detection
• Coevolutionary Defense