Cryptanalytic Extraction of Isolated Bias-Free GLU Feed-Forward Blocks by Antipodal Separation

2026-08-06

AI Summary

Existing cryptanalytic approaches struggle to effectively extract internal parameters from the bias-free Gated Linear Unit (GLU) feedforward modules widely adopted in modern language models. This work proposes a multi-stage forward-query recovery method that combines finite-difference curvature analysis with observations from antipodal input pairs (x and −x) to decouple and accurately reconstruct the gating direction, magnitude, and value branches. It presents the first constructive extraction of an isolated bias-free GLU module, overcoming prior limitations that applied only to single-branch activation functions (e.g., ReLU, GELU) or final projection layers. The approach achieves median validation errors below 1% across high-dimensional settings—including a six-layer Qwen model, an 8192-dimensional Llama subnetwork, and full-dimensional Gemma—and maintains errors under 5% across four finite-precision configurations.

Abstract

Cryptanalytic extraction has been demonstrated for ReLU networks, for networks using componentwise activations such as GELU or SiLU, and for a Transformer's final projection matrix. These methods do not recover the bias-free Gated Linear Unit (GLU) feed-forward blocks used in many modern language models. Such a block multiplies an activated linear projection by a second learned linear projection within each hidden unit, a two-branch structure absent from the network classes and final-layer setting addressed by those methods. We give a constructive, multi-stage forward-query recovery primitive for isolated bias-free GLU blocks. Finite-difference curvature supplies gate-direction candidates, and paired observations at x and -x separate gate magnitude, orientation, and value-branch coupling. Across high-precision targets, six Qwen layers, an 8,192-unit Llama subproblem, and a full-dimensional Gemma block all reach sub-percent median validation error. Four finite-precision configurations remain below 5 percent median error, but none reproduces every stored weight. These isolated-block experiments are not an end-to-end model-API attack: deriving the required internal block responses from final model outputs remains unsolved.

Problem

• Cryptanalytic extraction
• Gated Linear Unit
• bias-free
• feed-forward blocks
• Transformer

Innovation

• GLU
• cryptanalytic extraction
• antipodal separation
• bias-free feed-forward block
• finite-difference curvature