Multi-Level Modeling of Large Language Model Inference Latency and Energy via Hybrid Analytical--Machine-Learning Predictors

2026-08-06

AI Summary

This work addresses the challenges of high computational cost, substantial energy consumption, and unpredictable latency in large language model (LLM) inference, which hinder sustainable deployment and hardware-aware design. To overcome these limitations, the authors propose HYMELL, a novel framework that uniquely integrates analytical modeling with machine learning within a three-tier hierarchical structure. This architecture models elementary operators, high-level components—including multi-head attention (MHA), grouped-query attention (GQA), and mixture-of-experts (MoE)—as well as system-level overheads across both prefill and decoding phases. Notably, HYMELL enables accurate prediction of inference latency and energy consumption using only architectural parameters, eliminating the need for empirical hardware measurements. Evaluation on NVIDIA H100 with LLaMA-3 8B demonstrates prediction errors below 5%, significantly enhancing both design space exploration efficiency and prediction accuracy.

Abstract

The rapid scaling of Large Language Models (LLMs) has significantly increased computational cost, energy consumption, and inference latency, making accurate estimation essential for sustainable artificial intelligence deployment and hardware-aware design. In this work, we introduce Hybrid Modeling for Energy and Latency of LLMs (HYMELL), a hybrid three-level framework for estimating LLM inference latency and energy by combining analytical modeling with machine learning (ML). HYMELL models LLM execution through a three-level hierarchy: analytical estimation of primitive operations, ML prediction of higher-level components, and an end-to-end model that captures system-level overheads across both prefill and decode phases. The framework supports diverse architectures, including dense and mixture-of-experts (MoE) feed-forward networks (FFNs), as well as multi-head attention (MHA) and grouped-query attention (GQA) mechanisms. Evaluated on an NVIDIA H100 graphics processing unit (GPU), HYMELL achieves high predictive accuracy; notably, for LLaMA 3 8B, it attains less than 5% error for both prefill and decode phases. By predicting execution costs directly from architectural parameters, it enables fast, hardware-free design space exploration and energy-efficient optimization.

Problem

• Large Language Models
• Inference Latency
• Energy Consumption
• Computational Cost
• Hardware-aware Design

Innovation

• hybrid modeling
• LLM inference latency
• energy prediction
• analytical-machine learning
• hardware-aware design