🤖 AI Summary
This work proposes a lightweight, unsupervised prompt optimization framework based on pairwise preferences, addressing key limitations of existing methods that rely on annotated data, produce verbose and repetitive prompts, and require extensive manual tuning. The approach leverages only an initial prompt and natural language criteria, using a large language model as a discriminator to provide preference-based feedback for iterative refinement. This method significantly enhances prompt conciseness and diversity while reducing the risk of “cheating” through overfitting to evaluation metrics. Evaluated across nine Big-Bench Hard (BBH) tasks, it achieves state-of-the-art or comparable performance on six, and matches TextGrad’s results on IFEval-Hard. Moreover, it reduces prompt length and repetition by 3–5×, with both human and model-based evaluations consistently outperforming baseline methods.
📝 Abstract
Prompt engineering is effective but labor-intensive, motivating automated optimization methods. Existing methods typically require labeled datasets, which are often unavailable, and produce verbose, repetitive prompts. We introduce PrefPO, a minimal prompt optimization approach inspired by reinforcement learning from human feedback (RLHF). Its preference-based approach reduces the need for labeled data and hyperparameter tuning-only a starting prompt and natural language criteria are needed. PrefPO uses an LLM discriminator to express pairwise preferences over model outputs and provide feedback to an LLM optimizer, iteratively improving performance. We evaluate PrefPO on 9 BIG-Bench Hard (BBH) tasks and IFEval-Hard, a newly-curated, challenging subset of IFEval. PrefPO matches or exceeds SOTA methods, including GEPA, MIPRO, and TextGrad, on 6/9 tasks and performs comparably to TextGrad on IFEval-Hard (82.4% vs 84.5%). Unlike other methods, PrefPO can optimize in both labeled and unlabeled settings. Without labels, PrefPO closely matches its labeled performance on 6/9 tasks, proving effective without ground truth. PrefPO also improves prompt hygiene: we find existing methods produce prompts 14.7x their original length or with 34% repetitive content; PrefPO reduces these issues by 3-5x. Furthermore, both LLM and human judges rate PrefPO's prompts higher than TextGrad's. Finally, we identify prompt hacking in prompt optimizers, where methods game evaluation criteria, and find PrefPO is susceptible at half the rate of TextGrad (37% vs 86%), generating fewer brittle, misaligned prompts.