When Models Can't Follow: Testing Instruction Adherence Across 256 LLMs

2025-10-18

AI Summary

Existing LLM instruction-following evaluations are vulnerable to benchmark contamination, compromising their fidelity in assessing true model capabilities. To address this, we propose a lightweight, reproducible evaluation framework: a compact test suite comprising 20 verifiable instructions, systematically covering four core dimensions—format compliance, content constraints, logical ordering, and multi-step execution—while mitigating data contamination and computational overhead. We conduct systematic evaluation across 256 mainstream and emerging LLMs, enabling the first cross-model, multi-dimensional empirical analysis of instruction-following behavior. Our analysis uncovers prevalent failure patterns, identifying logical inconsistency and poor multi-step coordination as critical weaknesses. This work establishes the most comprehensive benchmark and diagnostic toolkit for LLM instruction-following capability to date, supporting rigorous, contamination-free assessment and targeted model improvement.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Search and Optimization: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Despite widespread deployment of Large Language Models, systematic evaluation of instruction-following capabilities remains challenging. While comprehensive benchmarks exist, focused assessments that quickly diagnose specific instruction adherence patterns are valuable. As newer models may be trained on existing benchmarks, novel evaluation approaches are needed to assess genuine capabilities rather than memorized performance. This paper presents a streamlined evaluation framework using twenty carefully designed prompts to assess LLM instruction-following across diverse task categories. We demonstrate this framework through a large-scale empirical study conducted on October 14, 2025, testing 256 verified working models from 331 available via OpenRouter. To ensure methodological rigor and prevent selection bias, we first verified each model's basic functionality before inclusion. Unlike large-scale benchmarks requiring extensive computational resources, our approach offers a practical diagnostic tool researchers and practitioners can readily apply. Our methodology builds upon verifiable instructions while introducing a compact test suite balancing comprehensiveness with efficiency. Each prompt targets distinct aspects of instruction following, including format compliance, content constraints, logical sequencing, and multi-step task execution. We evaluate models from major providers (OpenAI, Anthropic, Google, Meta, Mistral) and emerging implementations (Qwen, DeepSeek, community models), providing comparative performance analysis. Our findings reveal consistent failure modes and identify specific instruction types posing particular challenges. This work contributes both a practical evaluation tool and one of the most comprehensive empirical analyses of instruction-following capabilities across the contemporary LLM landscape.

Problem

• Testing instruction-following capabilities across 256 LLMs
• Developing streamlined evaluation framework for instruction adherence
• Identifying consistent failure modes in diverse instruction types

Innovation

• Compact test suite with twenty verifiable instruction prompts
• Methodology balances evaluation comprehensiveness with practical efficiency
• Framework assesses format compliance and multi-step execution