From Tokens to Watt-hours: Analytical Energy Estimation for LLM Inference on Modern GPUs

2026-07-29

AI Summary

Accurate energy estimation for large language model (LLM) inference remains challenging without hardware-level monitoring, hindering the advancement of green AI. This work proposes an analytical energy estimation method that operates without runtime measurements, leveraging model parameters, memory access patterns, and GPU hardware characteristics—such as NVIDIA H100’s FP16/BF16 Tensor Cores and HBM energy coefficients—to separately model energy consumption during the prefill and autoregressive decoding phases. The approach decomposes energy usage into fine-grained components including computation, parameter loading, KV cache writing, and attention reading. To the best of our knowledge, this is the first method to enable transparent, reproducible energy estimation across the entire LLM inference pipeline, facilitating energy-efficiency comparisons and design optimizations under varying model scales, context lengths, and generation lengths, thereby offering a practical tool for sustainable AI deployment.

Abstract

The operational energy consumption of large language model (LLM) inference is becoming an increasingly important component of the environmental footprint of deployed AI systems. However, direct measurement of inference energy often requires hardware telemetry, power instrumentation, or infrastructure-specific monitoring, limiting its applicability in comparative studies, early-stage system design, and sustainability reporting. This report presents an analytically structured, empirically calibrated, GPU-level methodology for estimating LLM inference energy on NVIDIA H100-class accelerators without direct runtime measurement. The proposed estimator combines parameter-scaled transformer FLOP accounting, calibrated memory-traffic factors, and hardware-specific energy coefficients for FP16/BF16 tensor-core computation and high-bandwidth-memory movement. It explicitly separates prompt prefill from autoregressive decoding, enabling energy estimates for input tokens, output tokens, and complete inference requests. The methodology further decomposes total energy into compute, parameter-access, key-value-cache write, and attention-read components, allowing the scaling behavior with model size, context length, and generated-token count to be analyzed. The resulting estimates are not intended to replace physical power measurements; rather, they provide transparent, reproducible, and assumption-explicit approximations suitable for model comparison, green-coding analysis, and design-time evaluation of LLM inference workloads.

Problem

• energy estimation
• LLM inference
• GPU
• sustainability
• large language models

Innovation

• energy estimation
• LLM inference
• GPU power modeling
• memory traffic
• transformer FLOPs