🤖 AI Summary
This paper addresses the lack of a systematic theoretical framework for automated prompt engineering by proposing the first unified optimization-theoretic survey across modalities. It formalizes discrete, continuous, and hybrid prompt variables—including instructions, soft prompts, and in-context examples—as constrained optimization problems, applicable to text, vision, and multimodal tasks. Methodologically, it integrates gradient-based optimization, evolutionary algorithms, and reinforcement learning to enable end-to-end, objective-driven prompt generation. Key contributions are: (1) the first optimization-theory-driven taxonomic framework for cross-modal prompt engineering; (2) the explicit identification and delineation of two frontier directions—constrained optimization and agent-oriented prompt design; and (3) a comprehensive knowledge system spanning the full technical spectrum and application domains of automated prompt engineering, providing a scalable theoretical foundation and methodological guidance for both research and practice.
📝 Abstract
The rise of foundation models has shifted focus from resource-intensive fine-tuning to prompt engineering, a paradigm that steers model behavior through input design rather than weight updates. While manual prompt engineering faces limitations in scalability, adaptability, and cross-modal alignment, automated methods, spanning foundation model (FM) based optimization, evolutionary methods, gradient-based optimization, and reinforcement learning, offer promising solutions. Existing surveys, however, remain fragmented across modalities and methodologies. This paper presents the first comprehensive survey on automated prompt engineering through a unified optimization-theoretic lens. We formalize prompt optimization as a maximization problem over discrete, continuous, and hybrid prompt spaces, systematically organizing methods by their optimization variables (instructions, soft prompts, exemplars), task-specific objectives, and computational frameworks. By bridging theoretical formulation with practical implementations across text, vision, and multimodal domains, this survey establishes a foundational framework for both researchers and practitioners, while highlighting underexplored frontiers in constrained optimization and agent-oriented prompt design.