In-Context and Few-Shots Learning for Forecasting Time Series Data based on Large Language Models

2025-12-08

AI Summary

This study systematically benchmarks large language models (LLMs) and pretrained time-series foundation models (e.g., TimesFM) against classical forecasting methods (ARIMA, LSTM, TCN, Transformer) to delineate their performance boundaries in time-series forecasting. Method: We propose a fine-tuning-free in-context learning (ICL) framework that unifies LLMs (o4-mini, Gemini 2.5 Flash Lite) and TimesFM into zero-shot and few-shot forecasting pipelines. Contribution/Results: Experimental results demonstrate that TimesFM achieves the best overall trade-off (RMSE = 0.3023, inference latency = 266 seconds), while o4-mini remains competitive in zero-shot settings. Crucially, this work validates the direct applicability of pretrained time-series foundation models to ICL—establishing a lightweight, efficient, and parameter-efficient forecasting paradigm that eliminates the need for task-specific fine-tuning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Existing data-driven approaches in modeling and predicting time series data include ARIMA (Autoregressive Integrated Moving Average), Transformer-based models, LSTM (Long Short-Term Memory) and TCN (Temporal Convolutional Network). These approaches, and in particular deep learning-based models such as LSTM and TCN, have shown great results in predicting time series data. With the advancement of leveraging pre-trained foundation models such as Large Language Models (LLMs) and more notably Google's recent foundation model for time series data, {it TimesFM} (Time Series Foundation Model), it is of interest to investigate whether these foundation models have the capability of outperforming existing modeling approaches in analyzing and predicting time series data. This paper investigates the performance of using LLM models for time series data prediction. We investigate the in-context learning methodology in the training of LLM models that are specific to the underlying application domain. More specifically, the paper explores training LLMs through in-context, zero-shot and few-shot learning and forecasting time series data with OpenAI { t o4-mini} and Gemini 2.5 Flash Lite, as well as the recent Google's Transformer-based TimesFM, a time series-specific foundation model, along with two deep learning models, namely TCN and LSTM networks. The findings indicate that TimesFM has the best overall performance with the lowest RMSE value (0.3023) and the competitive inference time (266 seconds). Furthermore, OpenAI's o4-mini also exhibits a good performance based on Zero Shot learning. These findings highlight pre-trained time series foundation models as a promising direction for real-time forecasting, enabling accurate and scalable deployment with minimal model adaptation.

Problem

• Evaluating LLMs for time series forecasting accuracy
• Comparing in-context and few-shot learning methods
• Assessing foundation models against traditional deep learning approaches

Innovation

• Uses in-context learning with LLMs for time series
• Compares foundation models like TimesFM against traditional methods
• Demonstrates TimesFM's superior accuracy and competitive speed