The Impact of Role Design in In-Context Learning for Large Language Models

2025-09-27

AI Summary

This study investigates the impact of role prompting on large language models’ (LLMs) in-context learning (ICL) performance, focusing on zero-shot and few-shot settings. We propose a structured role prompting methodology and systematically evaluate its efficacy across four task categories—sentiment analysis, text classification, question answering, and mathematical reasoning—using GPT-3.5, GPT-4o, Llama2-7b, and Llama2-13b. Results demonstrate that judicious incorporation of role information consistently enhances model generalization and task adaptation across diverse models and tasks, without requiring fine-tuning. Our core contribution is the first empirical identification of role design as an independent, highly effective, and controllable intervention dimension in ICL. This finding establishes a theoretical foundation and practical framework for interpretable, reusable prompt engineering—advancing lightweight, parameter-free optimization strategies for LLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Prompt Engineering / Prompting
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

In-context learning (ICL) enables Large Language Models (LLMs) to generate predictions based on prompts without additional fine-tuning. While prompt engineering has been widely studied, the impact of role design within prompts remains underexplored. This study examines the influence of role configurations in zero-shot and few-shot learning scenarios using GPT-3.5 and GPT-4o from OpenAI and Llama2-7b and Llama2-13b from Meta. We evaluate the models' performance across datasets, focusing on tasks like sentiment analysis, text classification, question answering, and math reasoning. Our findings suggest the potential of role-based prompt structuring to enhance LLM performance.

Problem

• Examining role design impact in in-context learning
• Evaluating role configurations across multiple LLMs and tasks
• Investigating role-based prompts to enhance model performance

Innovation

• Role-based prompt structuring enhances LLM performance
• Evaluates role configurations in zero-shot and few-shot learning
• Tests GPT and Llama models across multiple tasks