SiLQ: Simple Large Language Model Quantization-Aware Training

2025-07-22

AI Summary

To address substantial accuracy degradation, high training overhead, and poor compatibility with specialized inference accelerators in large language model (LLM) quantization, this paper proposes a lightweight end-to-end quantization-aware training (QAT) method. Our approach introduces no auxiliary operators and uniformly quantizes weights, activations, and KV cache to low-bit precision—preserving the original model architecture and full hardware compatibility. It incurs less than 0.1% additional training cost while enabling full-network parameter quantization. Evaluated across multiple mainstream benchmarks, our method consistently outperforms state-of-the-art (SOTA) quantization techniques: both base and instruction-tuned models sustain minimal accuracy loss (average <0.5%), achieve 4× model size reduction, deliver 30–50% lower inference latency, and reduce energy consumption by approximately 40%.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Natural Language Processing: (Large) Language ModelsComputer Vision: Large Vision Models

Application Category

User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendationEconomics, Online Markets and Human Computation: Cost models of using LLMs in production systemsSearch and Retrieval-Augmented AI: Large language models for search

Abstract

Large language models can be quantized to reduce inference time latency, model size, and energy consumption, thereby delivering a better user experience at lower cost. A challenge exists to deliver quantized models with minimal loss of accuracy in reasonable time, and in particular to do so without requiring mechanisms incompatible with specialized inference accelerators. Here, we demonstrate a simple, end-to-end quantization-aware training approach that, with an increase in total model training budget of less than 0.1%, outperforms the leading published quantization methods by large margins on several modern benchmarks, with both base and instruct model variants. The approach easily generalizes across different model architectures, can be applied to activations, cache, and weights, and requires the introduction of no additional operations to the model other than the quantization itself.

Problem

• Minimize accuracy loss in quantized large language models
• Achieve quantization without incompatible inference accelerator mechanisms
• Generalize quantization across model architectures and components

Innovation

• Simple quantization-aware training for large models
• Minimal accuracy loss with 0.1% training overhead
• Generalizes across architectures and model components