TimeSynth: A Framework for Uncovering Systematic Biases in Time Series Forecasting

2026-02-11

AI Summary

This work addresses the limitations of existing time series forecasting benchmarks, which often exhibit simplistic dynamics and suffer from data-dependent evaluation biases that can misrepresent model performance. To overcome these issues, we propose TimeSynth, the first structured synthetic framework that generates signals incorporating real-world characteristics such as non-stationarity, trends, periodicity, and phase modulation. By calibrating distributional shifts and noise levels based on parameters derived from real data, TimeSynth enables a systematic evaluation of diverse models—including linear methods, MLPs, CNNs, and Transformers—under complex temporal dynamics. Our experiments reveal that linear models degrade to simple oscillatory behavior under such complexity, whereas Transformers and CNNs demonstrate significantly superior adaptability and robustness.

Technology Category

• Machine Learning: Time-Series/Data Streams
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Models for Web evolution
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Time series forecasting is a fundamental tool with wide ranging applications, yet recent debates question whether complex nonlinear architectures truly outperform simple linear models. Prior claims of dominance of the linear model often stem from benchmarks that lack diverse temporal dynamics and employ biased evaluation protocols. We revisit this debate through TimeSynth, a structured framework that emulates key properties of real world time series,including non-stationarity, periodicity, trends, and phase modulation by creating synthesized signals whose parameters are derived from real-world time series. Evaluating four model families Linear, Multi Layer Perceptrons (MLP), Convolutional Neural Networks (CNNs), and Transformers, we find a systematic bias in linear models: they collapse to simple oscillation regardless of signal complexity. Nonlinear models avoid this collapse and gain clear advantages as signal complexity increases. Notably, Transformers and CNN based models exhibit slightly greater adaptability to complex modulated signals compared to MLPs. Beyond clean forecasting, the framework highlights robustness differences under distribution and noise shifts and removes biases of prior benchmarks by using independent instances for train, test, and validation for each signal family. Collectively, TimeSynth provides a principled foundation for understanding when different forecasting approaches succeed or fail, moving beyond oversimplified claims of model equivalence.

Problem

• time series forecasting
• systematic bias
• evaluation protocol
• model comparison
• temporal dynamics

Innovation

• TimeSynth
• systematic bias
• time series forecasting
• nonlinear models
• synthetic benchmark