🤖 AI Summary
This study addresses the challenge of precisely quantifying the environmental impact of large model inference in cloud environments by proposing a systematic evaluation method that separately measures the energy consumption, water usage, and carbon emissions associated with input and output tokens. By integrating GPU benchmarking, Bayesian linear regression, and Monte Carlo simulation, this work pioneers a reliable estimation of energy consumption uncertainty for both closed-source and open-source models with unknown configurations. The proposed framework provides actionable quantitative evidence to support the reduction of cloud service costs, electricity consumption, and carbon footprints.
📝 Abstract
We describe a methodology for estimating the per-token energy cost of cloud-hosted large language model (LLM) inference, separating between input (prefill) and output (decode) tokens. Graphics processing unit (GPU) energy usage is measured during inference benchmarking with open-weights models on a wide range of text-based tasks. The remaining server energy contribution from non-GPU hardware is estimated from the inference wall time. Bayesian linear regression is used to model the relationship between energy per token and LLM size, request traffic, and hardware deployment configuration. Proprietary frontier LLMs of unknown size and deployment are binned into size buckets based on naming conventions and performance priors, and the space of possible LLM configurations is sampled with Monte-Carlo methods to give a representative average energy per token and uncertainty. We also describe how these energy measurements can be used to estimate the carbon-dioxide equivalent ($\mathrm{CO_2\text{-}eq}$) emissions, both usage and embodied, and water consumed per token of AI inference. This methodology provides actionable data that enables reductions in cost, electricity usage, $\mathrm{CO_2\text{-}eq}$ emitted and water consumed in cloud and Software as a Service (SaaS).