Generation through the lens of learning theory

2024-10-17

arXiv.org

AI Summary

This paper investigates the theoretical foundations of generative capability, formalizing necessary and sufficient conditions for decidability of generation—unified, non-uniform, and prompt-based—and elucidating its fundamental relationship with predictability. Building on statistical learning theory, we introduce *closure dimension*—a novel combinatorial measure—as the core criterion characterizing generative capacity, thereby establishing a general framework for generability. We rigorously prove that generability and predictability (in both PAC and online learning models) are mutually exclusive. We provide complete, precise decidability characterizations—necessary and sufficient conditions—for all three generation paradigms. Furthermore, we unify and extend the framework of Kleinberg & Mullainathan (2024), modeling generation as a formal language closure problem over binary hypothesis classes. Our results bridge foundational learning theory with modern generative modeling, offering rigorous criteria to distinguish generative from predictive computation.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Deep Generative Models & Autoencoders
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We study generation through the lens of statistical learning theory. First, we abstract and formalize the results of Gold [1967], Angluin [1979], Angluin [1980] and Kleinberg and Mullainathan [2024] in terms of a binary hypothesis class defined over an abstract example space. Then, we extend the notion of"generation"from Kleinberg and Mullainathan [2024] to two new settings, we call"uniform"and"non-uniform"generation, and provide a characterization of which hypothesis classes are uniformly and non-uniformly generatable. As is standard in learning theory, our characterizations are in terms of the finiteness of a new combinatorial dimension termed the Closure dimension. By doing so, we are able to compare generatability with predictability (captured via PAC and online learnability) and show that these two properties of hypothesis classes are incompatible -- there are classes that are generatable but not predictable and vice versa. Finally, we extend our results to capture prompted generation and give a complete characterization of which classes are prompt generatable, generalizing some of the work by Kleinberg and Mullainathan [2024].

Problem

• Learning Theory
• Generative Conditions
• Predictivity

Innovation

• Unified Generation
• Closure Dimension
• Prompt Generation