LLM Compression with Jointly Optimizing Architectural and Quantization choices

2026-06-02

AI Summary

This work addresses the challenge of deploying large language models under stringent memory and computational constraints. Existing compression approaches often fall short of optimal performance due to the decoupling of neural architecture search and quantization strategies, as well as limited search spaces. To overcome these limitations, the paper proposes a differentiable neural architecture search framework that jointly optimizes linear layer configurations and mixed-precision quantization schemes in an end-to-end manner over a complete search space—marking the first method to achieve such co-optimization. Experimental results demonstrate that the proposed approach achieves a 1.4× speedup in inference latency while preserving model accuracy, or alternatively, yields an average accuracy gain of 6% across seven benchmark tasks under identical latency constraints.

Abstract

Deploying large language models (LLMs) is challenging due to their significant memory and computational requirements. While some methods address this by developing small or tiny language models from scratch, these approaches demand extensive GPU training. Compressing pre-trained LLMs for edge devices offers a compelling alternative. Beyond pruning and quantization, Neural Architecture Search (NAS) enables effective compression, yet prior NAS approaches often limit the search space and decouple architecture from quantization. We introduce a differentiable NAS framework that explores the entire space and jointly optimizes architectural configurations alongside mixed-precision quantization for linear layers of LLMs. Experiments demonstrate superior accuracy-latency trade-offs: our models achieve up to 1.4x faster inference than sequential NAS-then-quantization baselines at comparable accuracy, or up to 6% higher average accuracy across seven reasoning tasks at equivalent latency.

Problem

• LLM Compression
• Neural Architecture Search
• Quantization
• Edge Deployment
• Accuracy-Latency Trade-off

Innovation

• Neural Architecture Search
• Mixed-Precision Quantization
• LLM Compression
• Joint Optimization
• Differentiable NAS