Paraphrase Types Elicit Prompt Engineering Capabilities

2024-06-28

Conference on Empirical Methods in Natural Language Processing

AI Summary

This study investigates how linguistic dimensions of prompt formulation—morphology, syntax, and lexis—affect large language model (LLM) task performance. Method: We conduct controlled experiments across 120 diverse tasks using five LLMs, rigorously matching confounding factors such as prompt length and lexical diversity. This enables the first fine-grained, linguistically grounded attribution analysis of prompt rewriting. Contribution/Results: Semantic-preserving rewrites at morphological and lexical levels yield the largest performance gains, revealing LLMs’ high sensitivity to surface-form variation. We propose a multidimensional prompt rewriting generation framework that integrates cross-model behavioral comparison and median gain statistics. Evaluated on Mixtral 8x7B and LLaMA-3-8B, it achieves median task performance improvements of +6.7% and +5.5%, respectively—demonstrating that linguistically informed rewriting systematically enhances prompt robustness and effectiveness.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Much of the success of modern language models depends on finding a suitable prompt to instruct the model. Until now, it has been largely unknown how variations in the linguistic expression of prompts affect these models. This study systematically and empirically evaluates which linguistic features influence models through paraphrase types, i.e., different linguistic changes at particular positions. We measure behavioral changes for five models across 120 tasks and six families of paraphrases (i.e., morphology, syntax, lexicon, lexico-syntax, discourse, and others). We also control for other prompt engineering factors (e.g., prompt length, lexical diversity, and proximity to training data). Our results show a potential for language models to improve tasks when their prompts are adapted in specific paraphrase types (e.g., 6.7% median gain in Mixtral 8x7B; 5.5% in LLaMA 3 8B). In particular, changes in morphology and lexicon, i.e., the vocabulary used, showed promise in improving prompts. These findings contribute to developing more robust language models capable of handling variability in linguistic expression.

Problem

• Language Models
• Prompting Strategies
• Performance Optimization

Innovation

• Prompt Engineering
• Language Model Performance
• Expression Diversity