ACING: Actor-Critic for Instruction Learning in Black-Box Large Language Models

2024-11-19

arXiv.org

AI Summary

Instruction optimization for black-box large language models (LLMs)—improving prompt quality without access to model parameters or gradients—remains a fundamental challenge. Method: We propose the first actor-critic–based, gradient-free reinforcement learning framework for this setting. We formulate instruction optimization as a stateless, continuous-action bandit problem, enabling non-differentiable, reward-driven prompt search. Our approach employs non-parametric reward modeling and API-level interactive training, eliminating reliance on internal model architecture. Contribution/Results: Evaluated on 30 instruction-following tasks using ChatGPT, our method achieves a median performance gain of 10 percentage points and outperforms human-crafted expert instructions by up to 39 points. To our knowledge, this is the first work to empirically validate the effectiveness and superiority of continuous-policy optimization in black-box prompt engineering.

Technology Category

• Search and Optimization: Learning to Search
• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Reinforcement Learning

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The effectiveness of Large Language Models (LLMs) in solving tasks vastly depends on the quality of the instructions, which often require fine-tuning through extensive human effort. This highlights the need for automated instruction optimization; however, this optimization is particularly challenging when dealing with black-box LLMs, where model parameters and gradients remain inaccessible. We propose ACING, a task-specific prompt optimization approach framed as a stateless continuous-action Reinforcement Learning (RL) problem, known as the continuum bandit setting. ACING leverages an actor-critic-based method to optimize prompts, learning from non-differentiable reward signals. We validate ACING by optimizing prompts for ChatGPT on 30 instruction-based tasks. ACING consistently outperforms baseline methods, achieving a median score improvement of 10 percentage points. Furthermore, ACING not only recovers but also surpasses human-crafted expert instructions, achieving up to a 39 percentage point improvement against human benchmarks.

Problem

• Optimizing instructions for black-box LLMs without gradients
• Automating instruction design to reduce human effort
• Exploring infinite instruction spaces using reinforcement learning

Innovation

• Actor-critic reinforcement learning for instruction optimization
• Explores infinite instruction spaces with black-box feedback
• Stateless continuous-action formulation for automated prompt discovery