Olivia: Harmonizing Time Series Foundation Models with Power Spectral Density

Unknown Date

AI Summary

This work addresses the challenge of limited representation transferability in time series foundation models when pretrained on heterogeneous cross-domain data, primarily due to divergent temporal patterns. To overcome this, the study introduces normalized power spectral density (PSD) as a spectral alignment prior and proposes a Harmonizer module that implicitly aligns PSD distributions across datasets. Building upon this, a low-dimensional resonant interaction space is constructed to enable an efficient attention mechanism—HarmonicAttention—that jointly shares and reparameterizes second-order temporal correlations. Evaluated across eight datasets from TSLib, GIFT-Eval, and GluonTS, the method consistently achieves state-of-the-art performance in zero-shot, few-shot, and full-data forecasting scenarios.

Abstract

Time series foundation models rely on large-scale pretraining over diverse datasets across domains, yet their heterogeneity in temporal patterns could hinder the effectiveness of training and learning transferable time series representations. Inspired a fundamental concept, normalized power spectral density (PSD) in signal processing, we assume harmonizing datasets via PSDs in the spectral domain could reduce mismatches and enhance pretraining. We then go beyond the direct intractable minimization optimization and innovatively reformulate it as a principled harmonization approach. Specifically, we propose Harmonizer, a module that reshapes spectral structures and implicitly harmonizing PSDs across datasets, which theoretically corresponds to a shared reparameterization of second-order temporal correlations. Our theoretical analysis further reveals token interactions with Harmonizer can be efficiently mediated by a compact set of resonators, motivating a HarmonicAttention design that performs self-attention in a low-dimensional interaction space. Then, we propose Olivia, a novel time series foundation model built upon these harmonization mechanisms. Extensive experiments on two large-scale benchmarks (TSLib and GIFT-Eval) and extra 6 datasets from GluonTS, demonstrate Olivia consistently achieves state-of-the-art performance under zero-shot, few-shot, and full-shot forecasting scenarios. Our code is available at https://github.com/TSTS13/Olivia.

Problem

• time series foundation models
• temporal heterogeneity
• pretraining
• transferable representations
• power spectral density

Innovation

• Power Spectral Density
• Time Series Foundation Model
• HarmonicAttention
• Spectral Harmonization
• Reparameterization