WinoTS: Wavelet-based Self-Distillation for Time Series Models

2026-09-30

AI Summary

This study addresses the bottleneck in time-series self-supervised pretraining where high-frequency noise disrupts structural learning and visual augmentation techniques are difficult to transfer directly. To overcome this, we propose a novel time-frequency self-distillation framework tailored for temporal signals. Centered on the wavelet transform, our method constructs invariant views through multi-scale time-frequency augmentations rather than conventional spatial transformations, enabling architecture-agnostic and efficient representation learning. Extensive experiments demonstrate that the proposed model surpasses state-of-the-art methods across long-term forecasting, zero-shot transfer, and anomaly detection tasks. Notably, linear probing on frozen representations outperforms fully supervised baselines, validating both the effectiveness and generalization capability of the proposed paradigm.

Abstract

Self-supervised pre-training of time series models is currently dominated by next-token prediction and reconstruction objectives. In continuous-valued domains, these paradigms often waste model capacity on high-frequency, point-wise noise at the expense of learning invariant structure. While invariance-based self-distillation has proven highly effective in computer vision, its application to temporal data remains largely underexplored. Effectively adapting such methods to time series requires carefully designed augmentations: spatial operations like cropping can shift the timing of repeating cycles or distort the signal, while basic jittering may provide limited variation. We introduce Wavelet-based self-distillation for time series (WinoTS), an invariance-based pre-training paradigm designed specifically for temporal signals. At its core, WinoTS leverages time-frequency augmentations to construct multi-scale structural views without distorting underlying signal dynamics. Across extensive evaluations, WinoTS outperforms state-of-the-art baselines in long-term forecasting, cross-domain zero-shot transfer, and unsupervised anomaly detection. Notably, linear probing on frozen WinoTS representations frequently surpasses fully supervised models trained from scratch. Systematic ablations demonstrate that WinoTS is a flexible, architecture-agnostic framework yielding gains across time series backbones, and establish that time-frequency transformations provide a principled alternative to vision-style spatial augmentations.

Problem

• time series
• self-supervised pre-training
• self-distillation
• invariance learning
• data augmentation

Innovation

• Self-distillation
• Wavelet transform
• Time-frequency augmentation
• Time series pre-training
• Invariance learning