🤖 AI Summary
This study addresses the instability of large language models in adhering to user instructions when those instructions conflict with patterns established in prior dialogue history. It systematically uncovers, for the first time, the tension between instruction following and pattern completion by evaluating 13 models across 16 instruction types over up to 50 conversational turns using adversarial dialogues. The findings reveal dramatic fluctuations in instruction-following rates (ranging from 1% to 99%), uncorrelated with standard capability benchmarks. Output diversity—particularly in multi-token generation—significantly enhances robustness. Chain-of-thought reasoning frequently exhibits inconsistencies between internal reasoning and final outputs. Although models can predict their own behavior with 83.5% accuracy, they consistently underestimate their resistance to adversarial prompting.
📝 Abstract
Language models are trained to follow instructions, but they are also powerful pattern completers. What happens when these two objectives conflict? We construct conversations in which a user instruction to behave in a target way T (e.g., always output a specific token, answer in a particular language, or adopt a persona) is opposed by N hardcoded assistant turns demonstrating a competing pattern P. We then measure instruction-following (IF) rates in this setting, across 13 models and 16 different instructions, for up to 50 turns. Average instruction-following rates range from 1% to 99% across models, largely uncorrelated with standard capability benchmarks. The transition from instruction-following to pattern-following is universal but highly model-dependent. Robustness is modulated both by instruction content, with models resisting induction longer when instructions align with their trained value priors, and by output format, with diverse multi-token responses proving substantially more resistant than single-token outputs. Chain-of-thought reasoning improves robustness but does not eliminate susceptibility, and can produce dissociation between correct deliberation and incorrect output. When asked to predict their behavior in this setting, models achieve 83.5% accuracy on average but systematically underestimate their own resistance to induction pressure. These results suggest that instruction-following remains brittle under induction pressure even for otherwise capable models, and that output diversity, rather than semantic engagement with the input, is the primary factor predicting robustness.