Prox: Training-Free FFN Activation Sparsity via Approximate Intermediate-Channel Salience in LLMs

2026-07-29

AI Summary

This work addresses the significant performance degradation of large language models under high sparsity when using existing training-free activation sparsification methods, which suffer from limited channel selection strategies. To overcome this, the authors propose Prox, a novel framework that constructs shared masks by ranking the magnitudes of intermediate activations in SwiGLU layers—leveraging ordinal information rather than exact values—and approximates channel salience through an efficient two-stage, training-free pipeline. Prox avoids costly dense computations, inherently supports quantization and sparse attention, and integrates input sparsity, quantized proxy weights, and sparse feedforward networks for highly efficient inference. Evaluated across ten models spanning six architectures, Prox consistently outperforms current training-free baselines, achieving up to 1.99× end-to-end decoding speedup at 70% feedforward network sparsity.

Abstract

Feed-forward networks (FFNs) dominate memory traffic and computation in large language model (LLM) inference, making them a primary target for activation sparsification. However, existing training-free methods suffer substantial model-quality degradation at high sparsity due to limitations in their channel-selection strategies. We observe that the SwiGLU intermediate state provides a highly effective channel-selection signal, but obtaining it requires costly dense computation. To address this, we present \emph{Prox}, a two-stage training-free framework for sparse SwiGLU FFNs. Prox hinges on the key insight: sparse execution requires only the channel mask induced by the intermediate state, which can be constructed from the magnitude ranking of its entries rather than their exact values. Specifically, Stage 1 uses input sparsity and quantized proxy weights to construct a shared mask; Stage 2 computes the selected channels exactly, enabling sparse execution of all three projections. Across ten LLMs from six model families, Prox outperforms training-free baselines at all sparsity levels, achieves up to a $1.99\times$ end-to-end decoding speedup at 70\% FFN sparsity, and is compatible with quantization and sparse attention.

Problem

• activation sparsity
• feed-forward networks
• large language models
• channel selection
• model-quality degradation

Innovation

• activation sparsity
• training-free
• SwiGLU
• intermediate-channel salience
• sparse inference