🤖 AI Summary
Large language models (LLMs) lack gradient-based parameter updates and rely on in-context learning (ICL) and meta-prompting—yet no rigorous theoretical framework exists to formalize their semantics or behavior. Method: This work introduces the first unified formal framework grounded in category theory, rigorously modeling the semantic structure and behavioral properties of ICL and meta-prompting via categorical constructions, formal semantic analysis, and empirical validation. Contribution/Results: We formally establish the task-agnostic nature of meta-prompting and prove equivalence among mainstream meta-prompting methods. Experiments demonstrate that meta-prompting consistently outperforms standard prompting, yielding significant improvements in output controllability and cross-task generalization. This study provides the first provably sound theoretical foundation for prompt-based adaptation in LLMs without parameter updates.
📝 Abstract
Modern large language models (LLMs) are capable of interpreting input strings as instructions, or prompts, and carry out tasks based on them. Unlike traditional learners, LLMs cannot use back-propagation to obtain feedback, and condition their output in situ in a phenomenon known as in-context learning (ICL). Many approaches to prompting and pre-training these models involve the automated generation of these prompts, also known as meta-prompting, or prompting to obtain prompts. However, they do not formally describe the properties and behavior of the LLMs themselves. We propose a theoretical framework based on category theory to generalize and describe ICL and LLM behavior when interacting with users. Our framework allows us to obtain formal results around task agnosticity and equivalence of various meta-prompting approaches. Using our framework and experimental results we argue that meta-prompting is more effective than basic prompting at generating desirable outputs.