Do LLMs Adhere to Label Definitions? Examining Their Receptivity to External Label Definitions

2025-09-02

AI Summary

This study investigates whether large language models (LLMs) prioritize externally provided explicit label definitions or rely predominantly on internal parametric knowledge during reasoning. To address this, we conduct controlled experiments across general-purpose benchmarks (e.g., BoolQ, MultiRC) and domain-specific benchmarks (e.g., MedQA, SciTail), systematically evaluating model adherence to human-annotated versus LLM-generated label definitions. Results reveal that external definition integration is neither robust nor consistent: in general tasks, models heavily default to internal representations, whereas in domain-specific tasks, external definitions improve accuracy (by +2.1–5.7%) and enhance decision interpretability. We identify, for the first time, a “task-sensitivity” phenomenon in definition adoption, propose a novel quantitative metric for measuring definition adherence, and underscore the critical importance of modeling knowledge fusion mechanisms to achieve controllable, reliable reasoning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Common-Sense Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Do LLMs genuinely incorporate external definitions, or do they primarily rely on their parametric knowledge? To address these questions, we conduct controlled experiments across multiple explanation benchmark datasets (general and domain-specific) and label definition conditions, including expert-curated, LLM-generated, perturbed, and swapped definitions. Our results reveal that while explicit label definitions can enhance accuracy and explainability, their integration into an LLM's task-solving processes is neither guaranteed nor consistent, suggesting reliance on internalized representations in many cases. Models often default to their internal representations, particularly in general tasks, whereas domain-specific tasks benefit more from explicit definitions. These findings underscore the need for a deeper understanding of how LLMs process external knowledge alongside their pre-existing capabilities.

Problem

• Examining LLMs' adherence to external label definitions
• Assessing integration of external vs parametric knowledge
• Testing definition receptivity across domain-specific tasks

Innovation

• Controlled experiments across multiple explanation datasets
• Testing expert-curated and LLM-generated label definitions
• Analyzing integration of external versus internal knowledge