🤖 AI Summary
This study investigates the fundamental limits of language models in learning the true data-generating process of natural language solely from observed text sequences, particularly when unobserved contextual factors—such as facts, intentions, or social settings—influence generation. By distinguishing between the full conditional generative process, the marginalized text-only process, and the distribution learned by the model, the work introduces criteria based on local sufficient statistics and conditional mutual information to delineate when next-token prediction remains valid. It demonstrates that standard training implicitly assumes stationarity and ergodicity, assumptions often violated in heterogeneous corpora, and proves that marginal models succeed only when observed prefixes are approximately sufficient for latent variables. The paper further interprets retrieval-augmented generation (RAG) and tool use as mechanisms that restore conditional sufficiency, thereby transcending conventional language modeling paradigms.
📝 Abstract
Language models trained on observed sequences are often described as learning the conditional distribution of the next token given previous tokens. This description is only conditionally correct. A model trained on realized token trajectories does not observe full conditional laws; it receives sampled continuations. Moreover, real language generation is conditioned not only on previous words but also on non-textual circumstances: facts, events, intentions, goals, beliefs, social context, and task-specific constraints. This paper distinguishes three objects that are often conflated: the full conditional language process conditioned on latent circumstances, the marginal text-only process obtained by integrating those circumstances out, and the model-induced distribution learned from finite observed corpora.
The paper argues that interpreting model training as estimating the marginal text-only law requires strong assumptions of stationarity, representativeness, and ergodicity, assumptions that are standard in statistical estimation but problematic when applied to heterogeneous language corpora. Even if these assumptions hold, the marginal text-only law is useful only when the observed prefix is an approximately sufficient statistic for the latent circumstances relevant to continuation. In information-theoretic terms, usefulness requires that the residual conditional mutual information between the next token and the omitted circumstances, given the observed text, be small.
The paper then extends this argument to heterogeneous training corpora.
Finally, the paper interprets Retrieval Augmented Generation (RAG) and tool use as conditional sufficiency devices.