Learning vs Retrieval: The Role of In-Context Examples in Regression with LLMs

2024-09-06

arXiv.org

AI Summary

Existing studies lack a systematic characterization of in-context learning (ICL) mechanisms for regression tasks in large language models (LLMs), particularly regarding the trade-off between internal knowledge retrieval and context-based example learning. Method: The authors introduce the first regression-oriented ICL mechanism evaluation framework, integrating controlled regression datasets, attribution-aware prompt ablation, and quantitative attribution analysis across three mainstream LLMs. Contribution/Results: They empirically demonstrate that LLM regression behavior lies on a “retrieval–learning” continuum, with the dominant mode modulated significantly by task priors, example type, and information richness. Based on these findings, they propose task-aware prompting principles for regression. Results exhibit strong cross-model and cross-dataset robustness, providing both theoretical grounding and practical guidance for efficient regression prompt engineering.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Data Mining & Knowledge Management: Conversational Systems for Recommendation & RetrievalSearch and Optimization: Learning to Search

Application Category

Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved informationUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendationSemantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Generative Large Language Models (LLMs) are capable of being in-context learners. However, the underlying mechanism of in-context learning (ICL) is still a major research question, and experimental research results about how models exploit ICL are not always consistent. In this work, we propose a framework for evaluating in-context learning mechanisms, which we claim are a combination of retrieving internal knowledge and learning from in-context examples by focusing on regression tasks. First, we show that LLMs can solve real-world regression problems and then design experiments to measure the extent to which the LLM retrieves its internal knowledge versus learning from in-context examples. We argue that this process lies on a spectrum between these two extremes. We provide an in-depth analysis of the degrees to which these mechanisms are triggered depending on various factors, such as prior knowledge about the tasks and the type and richness of the information provided by the in-context examples. We employ three LLMs and utilize multiple datasets to corroborate the robustness of our findings. Our results shed light on how to engineer prompts to leverage meta-learning from in-context examples and foster knowledge retrieval depending on the problem being addressed.

Problem

• Mechanism of in-context learning in LLMs
• Retrieval vs learning in regression tasks
• Impact of example type on LLM performance

Innovation

• Evaluates in-context learning mechanisms
• Measures retrieval versus learning balance
• Engineers prompts for meta-learning