🤖 AI Summary
This study investigates whether the performance of large language models remains constant over time by conducting a longitudinal replication experiment on GPT-4o over three months, with ten evaluations every three hours under identical physics tasks. Employing time-series data collection, controlled experimental design, and Fourier spectral analysis, the research reveals—for the first time—significant diurnal and weekly periodic fluctuations in model performance, with approximately 20% of output variance attributable to these rhythmic patterns. These findings challenge the foundational assumption of temporal invariance in model performance, demonstrating that the output quality of large language models exhibits systematic time dependence. The results carry important implications for the reproducibility of AI research and the prevailing paradigms used to evaluate model capabilities.
📝 Abstract
Large language models (LLMs) are increasingly used in research both as tools and as objects of investigation. Much of this work implicitly assumes that LLM performance under fixed conditions (identical model snapshot, hyperparameters, and prompt) is time-invariant. If average output quality changes systematically over time, this assumption is violated, threatening the reliability, validity, and reproducibility of findings. To empirically examine this assumption, we conducted a longitudinal study on the temporal variability of GPT-4o's average performance. Using a fixed model snapshot, fixed hyperparameters, and identical prompting, GPT-4o was queried via the API to solve the same multiple-choice physics task every three hours for approximately three months. Ten independent responses were generated at each time point and their scores were averaged. Spectral (Fourier) analysis of the resulting time series revealed notable periodic variability in average model performance, accounting for approximately 20% of the total variance. In particular, the observed periodic patterns are well explained by the interaction of a daily and a weekly rhythm. These findings indicate that, even under controlled conditions, LLM performance may vary periodically over time, calling into question the assumption of time invariance. Implications for ensuring validity and replicability of research that uses or investigates LLMs are discussed.