GReaTer: Gradients over Reasoning Makes Smaller Language Models Strong Prompt Optimizers

2024-12-12

arXiv.org

AI Summary

Existing prompt optimization methods heavily rely on large language models’ (LLMs) textual feedback, rendering them incompatible with smaller models and overlooking exploitable gradient signals. This work proposes the first differentiable prompt optimization framework grounded in task-loss gradients: it formalizes the inference process as a differentiable computational graph and directly updates parameterized prompt embeddings via gradient descent, eliminating dependence on LLM-generated feedback. Its core innovation lies in introducing differentiable inference gradients into prompt engineering—enabling end-to-end, LLM-free prompt learning on resource-constrained models. Evaluated on BBH, GSM8K, and FOLIO, our method consistently surpasses state-of-the-art approaches. Moreover, the learned prompts exhibit strong generalization: on several tasks, performance matches or even exceeds that of significantly larger models.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Search and Optimization: Learning to Search
• Machine Learning: Optimization

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The effectiveness of large language models (LLMs) is closely tied to the design of prompts, making prompt optimization essential for enhancing their performance across a wide range of tasks. Many existing approaches to automating prompt engineering rely exclusively on textual feedback, refining prompts based solely on inference errors identified by large, computationally expensive LLMs. Unfortunately, smaller models struggle to generate high-quality feedback, resulting in complete dependence on large LLM judgment. Moreover, these methods fail to leverage more direct and finer-grained information, such as gradients, due to operating purely in text space. To this end, we introduce GReaTer, a novel prompt optimization technique that directly incorporates gradient information over task-specific reasoning. By utilizing task loss gradients, GReaTer enables self-optimization of prompts for open-source, lightweight language models without the need for costly closed-source LLMs. This allows high-performance prompt optimization without dependence on massive LLMs, closing the gap between smaller models and the sophisticated reasoning often needed for prompt refinement. Extensive evaluations across diverse reasoning tasks including BBH, GSM8k, and FOLIO demonstrate that GReaTer consistently outperforms previous state-of-the-art prompt optimization methods, even those reliant on powerful LLMs. Additionally, GReaTer-optimized prompts frequently exhibit better transferability and, in some cases, boost task performance to levels comparable to or surpassing those achieved by larger language models, highlighting the effectiveness of prompt optimization guided by gradients over reasoning. Code of GReaTer is available at https://github.com/psunlpgroup/GreaTer.

Problem

• Optimizing prompts without relying on large LLMs
• Leveraging gradients for self-optimization in small models
• Improving reasoning tasks performance with gradient-guided prompts

Innovation

• Uses gradient information for prompt optimization
• Enables self-optimization for smaller language models
• Eliminates dependence on large LLMs