TOTEM: TOkenized Time Series EMbeddings for General Time Series Analysis

2024-02-26

arXiv.org

AI Summary

This work addresses the general time-series analysis problem by proposing the first end-to-end discretization-based tokenization paradigm. The method unifies multi-source time-series data into discrete tokens via self-supervised vector quantization and contrastive learning, enabling zero-shot transfer across diverse tasks—imputation, anomaly detection, and forecasting—as well as cross-domain adaptation. It employs fixed token embeddings and a standard Transformer architecture, departing from conventional task- or dataset-specific modeling paradigms. Evaluated on nearly 500 experimental configurations across 12 real-world datasets, the model achieves state-of-the-art performance on all three core tasks, outperforming 17, 19, and 14 baselines in imputation, anomaly detection, and forecasting, respectively. Crucially, it maintains competitive accuracy under both specialized (task-tuned) and generalized (zero-shot) settings, significantly enhancing the generalizability and practical utility of foundational time-series models.

Technology Category

• Machine Learning: Time-Series/Data Streams
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: (Large) Language Models

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

This work studies the problem of time series analysis with generalist (or foundation) models, which are models trained across many data domains. Drawing inspiration from the widespread success of large language models, we consider the simple strategy of discretely tokenizing time series data drawn from a myriad of datasets via self-supervision, then using the fixed tokenization to solve a variety of tasks across many data domains. Canonically, time series models are either trained on a single dataset or built in a task-specific manner (e.g., a forecasting-only model), where many use patches of time as inputs to the model. As such, performant generalist, discrete representation time series models explored across many tasks are of value. Our method, TOkenized Time Series EMbeddings (TOTEM), produces such generalist time series models with minimal or no fine-tuning while exhibiting strong zero-shot performance. We evaluate TOTEM extensively over nearly 500 experiments on three commonly-studied time series tasks with real-world data: imputation (17 baselines, 12 datasets), anomaly detection (19 baselines, 25 datasets), and forecasting (14 baselines, 12 datasets). We conclude that TOTEM matches or outperforms existing state-of-the-art models in both the canonical specialist setting (i.e., training one model on one domain) as well as the generalist setting (i.e., training a single model on many domains), which demonstrates the efficacy of tokenization for general time series analysis. The open-source implementation is available here: https://github.com/SaberaTalukder/TOTEM; a video summary is available here: https://www.youtube.com/watch?v=OqrCpdb6MJk.

Problem

• Time Series Analysis
• Anomaly Detection
• Predictive Modeling

Innovation

• TOTEM
• Symbolic Representation
• Time Series Analysis