Scaling Point-in-Time Language Models

2026-04-24

AI Summary

This work addresses the issue of forward-looking bias in general-purpose large language models (LLMs), which arises from the inadvertent inclusion of future-dated information in training data and undermines the validity of backtesting and causal inference in finance and social sciences. To mitigate this, the authors propose Point-in-Time Language Models (PIT LMs), trained exclusively on temporally filtered FineWeb data comprising one trillion time-compliant tokens. They release monthly model snapshots from 2013 to 2024, each based on a 4-billion-parameter decoder-only Transformer architecture. Through large-scale pretraining and LoRA-based instruction tuning, PIT LMs substantially narrow the performance gap with temporally agnostic counterparts, achieving near-parity with Gemma-3-4B and LLaMA-7B on commonsense reasoning and language understanding benchmarks. The project provides fully open-source training and evaluation pipelines, establishing the first reproducible and temporally valid paradigm for large-model research.

Abstract

Large language models trained on unrestricted internet corpora inevitably embed information from the future, introducing lookahead bias that compromises the validity of backtests and causal inference in finance and the social sciences. Point-in-time language models--trained exclusively on text available up to each calendar date--eliminate this leakage by construction, but existing efforts typically produce models that lag substantially behind their unconstrained counterparts. We show that this performance gap can be substantially narrowed through scale. Training decoder-only transformers with up to 4 billion parameters on 1 trillion chronologically filtered tokens from FineWeb, we construct a sequence of monthly model checkpoints spanning 2013-2024. Across a range of common-sense reasoning and language understanding benchmarks, our models approach the performance of leading open-weight models of comparable size (e.g., Gemma-3-4B and LLaMA-7B) trained on temporally unrestricted data, although a performance gap remains on several tasks. Instruction fine-tuning via LoRA further improves downstream usability. We release the complete pipeline--including dataset construction, training infrastructure, and evaluation code--to enable reproducible point-in-time language modeling and to support research applications that require strict temporal validity.

Problem

• lookahead bias
• point-in-time language models
• temporal validity
• backtesting
• causal inference

Innovation

• point-in-time language models
• temporal validity
• lookahead bias
• chronologically filtered training
• LoRA fine-tuning