🤖 AI Summary
This work investigates the mechanism by which prompts induce behavioral switching in fixed-weight Transformers. Method: We propose the “Prompt-as-Program” theoretical framework, formalizing prompts as executable programs that leverage attention for memory routing, conditionally activate arithmetic operations in feed-forward networks (FFNs), and achieve multi-step compositional computation via depth-wise stacking—all atop a single frozen backbone. Using simplified Transformer modeling, mechanistic decomposition, and constructive existence proofs—grounded in computability and expressive power theory—we rigorously characterize capability boundaries and structural limits under constraints on prompt length and precision. Contribution/Results: This is the first unified formal analysis foundation for prompt engineering, transcending empirical practice and establishing a rigorous theoretical basis for prompt-driven computation in fixed-weight Transformers.
📝 Abstract
Prompts can switch a model's behavior even when the weights are fixed, yet this phenomenon is rarely treated as a clean theoretical object rather than a heuristic. We study the family of functions obtainable by holding a Transformer backbone fixed as an executor and varying only the prompt. Our core idea is to view the prompt as an externally injected program and to construct a simplified Transformer that interprets it to implement different computations. The construction exposes a mechanism-level decomposition: attention performs selective routing from prompt memory, the FFN performs local arithmetic conditioned on retrieved fragments, and depth-wise stacking composes these local updates into a multi-step computation. Under this viewpoint, we prove a constructive existential result showing that a single fixed backbone can approximate a broad class of target behaviors via prompts alone. The framework provides a unified starting point for formalizing trade-offs under prompt length/precision constraints and for studying structural limits of prompt-based switching, while remaining distinct from empirical claims about pretrained LLMs.