🤖 AI Summary
It remains unclear whether Transformers can algorithmically infer transition probabilities of Markov chains from context via next-token prediction alone—rather than relying on memorized patterns. Method: We train standard autoregressive Transformers on sequences generated by randomly sampled Markov chains with diverse transition matrices, systematically varying model size and training data scale. Contribution/Results: We identify dual thresholds in model capacity and data volume beyond which Transformers transition from pattern memorization to genuine context-driven estimation of transition probabilities. Incorporating explicit state embeddings significantly improves zero-shot generalization to unseen chain structures. Crucially, sufficiently large Transformers achieve accurate modeling of previously unobserved Markov chains without fine-tuning—demonstrating, for the first time, systematic evidence that Transformers can implicitly learn probabilistic algorithms from context. This provides foundational insight into the emergent algorithmic reasoning capabilities of large language models.
📝 Abstract
We investigate the capacity of transformers to learn algorithms involving their context while solely being trained using next token prediction. We set up Markov chains with random transition matrices and we train transformers to predict the next token. Matrices used during training and test are different and we show that there is a threshold in transformer size and in training set size above which the model is able to learn to estimate the transition probabilities from its context instead of memorizing the training patterns. Additionally, we show that more involved encoding of the states enables more robust prediction for Markov chains with structures different than those seen during training.