π€ AI Summary
This study investigates whether increasing the number of training tokens yields commensurate improvements in model performance and, for the first time, incorporates energy consumption and training duration into the evaluation framework. Under fixed hardware conditions, the authors conduct repeated experiments by training the 1.1B-parameter TinyLlama model on identical GPUs, combining high-frequency power sampling with repeated-measures ANOVA to systematically assess the joint impact of token scale on performance, energy use, and training time. The findings reveal diminishing returns in conventional performance metrics with increasing token counts, while energyβtime efficiency exhibits a strictly monotonic decline, indicating that high-token training can substantially reduce energy efficiency. This work introduces an energy-aware perspective on parameter efficiency, offering empirical foundations for greener large language model training.
π Abstract
Research in machine learning has questioned whether increases in training token counts reliably produce proportional performance gains in large language models. Building on prior work introducing an energy-aware parameter efficiency metric, this study empirically examines the effects of increasing training token counts under fixed hardware and training conditions. The significance of this work lies in the explicit integration of power consumption and execution duration, as reflected by the power sampling frequency, into token-scale analysis. This addresses a gap in prior studies emphasizing performance outcomes while underrepresenting computational and energy costs. Using a repeated-measures experimental design on a constant GPU instance with an identical model architecture, optimizer settings, and epoch counts, a 1.1-billion-parameter TinyLlama model was trained at three token counts (500K, 1M, and 2M). While conventional performance metrics exhibited inconsistent or diminishing returns across token scales, the inclusion of power consumption and execution duration revealed a strictly monotonic decline in training efficiency as token count increased. Repeated-measures ANOVA demonstrated a strong effect of token count on parameter efficiency, with all pairwise comparisons remaining significant following Bonferroni correction. These findings indicate that increases in training token counts may be energetically inefficient even when marginal performance improvements are observed, underscoring the importance of efficiency-aware evaluation in large language model training.