HADES: Hardware Accelerated Decoding for Efficient Speculation in Large Language Models

2024-12-27

AI Summary

To address high latency and energy consumption in large language model (LLM) inference caused by increasing model complexity, this paper proposes a hardware-native speculative decoding architecture. Unlike conventional software-level speculation, our approach deeply integrates speculative execution into the hardware accelerator design, enabling instruction-level speculative execution and overcoming synchronization and latency bottlenecks. The architecture comprises a RISC-V-based custom LLM acceleration core, a dedicated hardware predictor, a lightweight draft model co-scheduling circuit, and a dynamic cache coherence protocol. Evaluated on Llama-2-7B, the design achieves a 2.8× end-to-end throughput improvement, a 3.4× energy efficiency gain, and a 41% reduction in first-token latency—significantly outperforming state-of-the-art baselines including vLLM and SpecInfer.

Technology Category

Machine Learning: Hardware-aware MLNatural Language Processing: (Large) Language ModelsComputer Vision: Large Vision Models

Application Category

Economics, Online Markets and Human Computation: Cost models of using LLMs in production systemsSearch and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved informationUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) have revolutionized natural language processing by understanding and generating human-like text. However, the increasing demand for more sophisticated LLMs presents significant computational challenges due to their scale and complexity. This paper introduces Hardware Accelerated Decoding (HADES), a novel approach to enhance the performance and energy efficiency of LLMs. We address the design of an LLM accelerator with hardware-level speculative decoding support, a concept not previously explored in existing literature. Our work demonstrates how speculative decoding can significantly improve the efficiency of LLM operations, paving the way for more advanced and practical applications of these models.

Problem

• Large Language Models
• Computational Efficiency
• Energy Consumption

Innovation

• Hardware-Accelerated Decoding
• Large Language Models
• Energy Efficiency