Task Competence Is Not Instruction Following: Evaluating Instruction-Conflicting Behavior in Small Language Models

2026-07-21

AI Summary

This study investigates whether small-scale language models adhere to user instructions when those instructions conflict with their task capabilities—such as selecting incorrect answers or generating opposite sentiment—and reveals a decoupling between task proficiency and instruction following. To this end, the authors propose a cross-task evaluation paradigm for conflicting instructions and introduce the Instruction Following Failure Rate (IFFR) metric. Systematic experiments on the Qwen model series demonstrate that while smaller models retain task accuracy, they consistently disregard conflicting instructions, whereas larger models exhibit significantly stronger instruction-following behavior. This work provides the first quantitative evidence that task capability does not equate to controllable behavior, offering a novel perspective and methodology for evaluating model controllability.

Abstract

Instruction tuning is meant to make language models follow user requests, yet it is unclear whether small models comply when an instruction conflicts with their usual task behavior. We study this across three tasks - multiple-choice question answering (MCQA), sentiment classification, and mathematical question answering - by pairing a standard instruction with a conflicting non-standard one (select an incorrect option, output the opposite sentiment, or return twice the answer). This cross-task design allows us to test whether resistance to conflicting instructions is tied to specific task characteristics or reflects a broader behavioral tendency. As all predictions are scored against the original ground truth, a model that ignores the non-standard instruction still appears accurate. Using standard accuracy, non-standard accuracy, and an Instruction-Following Failure Rate (IFFR), we evaluate instruction-tuned Qwen models across sizes. Both standard accuracy and instruction following generally improve with scale, although the pattern is not consistent across all tasks and datasets. Small models stay competent yet routinely ignore the non-standard instruction, while larger models show a clear gap between the two settings. These findings suggest that gains in task capability do not automatically provide reliable control over model behavior. Task competence and instruction following are therefore distinct abilities, and reporting only standard accuracy hides instruction-following failures.

Problem

• instruction following
• task competence
• small language models
• instruction-conflicting behavior
• model evaluation

Innovation

• instruction following
• task competence
• conflicting instructions
• small language models
• Instruction-Following Failure Rate