instruction-following evaluation

This work addresses the growing risks of misuse and loss of control associated with the broad applicability of foundation models, which existing alignment methods struggle to mitigate through hard behavioral constraints. It establishes capability control as a core objective distinct from alignment and introduces a defense-in-depth framework spanning data, learning, and system layers to enforce multi-granular behavioral constraints throughout the model lifecycle. By integrating techniques such as data distribution shaping, representational intervention, and runtime input/output/action-level safeguards, the paper systematically constructs pathways for capability control. It further identifies critical challenges—including the dual-use nature of knowledge and combinatorial generalization—offering a new paradigm for developing safe and controllable AI systems.

Existing LLM instruction-following evaluations are vulnerable to benchmark contamination, compromising their fidelity in assessing true model capabilities. To address this, we propose a lightweight, reproducible evaluation framework: a compact test suite comprising 20 verifiable instructions, systematically covering four core dimensions—format compliance, content constraints, logical ordering, and multi-step execution—while mitigating data contamination and computational overhead. We conduct systematic evaluation across 256 mainstream and emerging LLMs, enabling the first cross-model, multi-dimensional empirical analysis of instruction-following behavior. Our analysis uncovers prevalent failure patterns, identifying logical inconsistency and poor multi-step coordination as critical weaknesses. This work establishes the most comprehensive benchmark and diagnostic toolkit for LLM instruction-following capability to date, supporting rigorous, contamination-free assessment and targeted model improvement.

This work investigates the mechanisms by which instruction tuning enhances general intelligence in Chinese large language models (LLMs), focusing on how data scale, model size (7B–33B), and data construction methodology (human-authored vs. synthetic) differentially affect multidimensional capabilities—including creative writing, code generation, and logical reasoning. Method: Leveraging a 40k+ multi-capability-annotated instruction dataset, we conduct cross-domain ablation studies to isolate these factors. Contribution/Results: We first reveal that underlying capabilities evolve at independent learning paces; human-authored data remains consistently effective, whereas synthetic data exhibits a performance ceiling; and instruction data demonstrates strong cross-capability generalization. Based on these findings, we propose a quantifiable, efficiency-oriented data construction guideline. Evaluated on two public benchmarks, our approach yields significant performance gains, providing empirical evidence and methodological foundations for capability-targeted LLM optimization.

This work addresses the weak instruction-following capability of large language models (LLMs) in knowledge-intensive tasks—e.g., failing to modify answers as instructed or being misled by irrelevant distractor instructions—by introducing the first verifiable instruction-following benchmark for knowledge tasks. Methodologically, it decouples instruction-following from factual knowledge by injecting conditional and distractor instructions into established knowledge benchmarks (e.g., MMLU, ARC); proposes an LLM-free automatic verification mechanism; designs two novel instruction paradigms—answer-dependent and option-space-driven; and conducts zero-shot evaluation across 25 open- and closed-source models spanning 1B to 405B parameters. Results reveal that even models extensively fine-tuned on instruction data frequently violate simple, unambiguous instructions. The project releases the benchmark dataset, evaluation code, and comprehensive results to support reproducible research.

This work challenges the implicit assumption in instruction tuning that “larger or stronger models necessarily make better teachers,” identifying and naming this phenomenon the *Large Model Paradox*. Through systematic evaluation of 20 response generators (teachers) across 5 base models, we find no monotonic positive correlation between teacher capability and student performance. To address this, we propose **Compatibility-Aware Reward (CAR)**—the first quantitative metric explicitly modeling teacher–base model compatibility, moving beyond conventional unidirectional quality-based evaluation (e.g., relying solely on teacher output quality). Via multi-model ablation, reward modeling, and empirical analysis, we demonstrate that CAR significantly outperforms existing metrics (e.g., ROUGE, BERTScore) in predicting teacher effectiveness and improving downstream instruction-following performance.

This work addresses the long-standing isolation among research domains such as alignment training, model organisms, and toy models, which has hindered empirical cross-pollination and led to redundant exploration and inefficiency. For the first time, it systematically transfers supervised fine-tuning (SFT) practices across these domains by integrating cross-model output training, mixed-strategy data, and benign fine-tuning to rigorously evaluate the portability of key findings. The study demonstrates three successful transfer effects: enhanced behavioral generalization, mitigation of capability degradation, and the critical insight that preserving capabilities alone is insufficient to ensure robustness in subsequent training phases. These results underscore both the efficacy and limitations of reusing methodologies across domains, thereby fostering more synergistic development across disparate research areas.

This work addresses the common oversimplification in distinguishing post-training methods for large language models—such as supervised fine-tuning (SFT) and reinforcement learning (RL)—by clarifying the distinction between “capability elicitation” and “capability creation.” Drawing on the free energy principle, the paper introduces the notion of an “accessible support set” and formalizes post-training as a reweighting of the pretraining distribution: reweighting within the support set corresponds to eliciting existing capabilities, whereas expanding the support set enables genuine capability creation. This framework transcends the conventional dichotomy based on SFT versus RL, revealing that both approaches primarily elicit rather than create capabilities under limited updates. It thus provides a novel theoretical foundation for understanding and designing post-training algorithms grounded in changes to behavioral reachability.

Existing benchmarks often conflate instruction following with task success, hindering accurate assessment of large language models’ true compliance capabilities under complex instructions. This work proposes MOSAIC, a modular framework that, for the first time, decomposes instruction compliance into independently analyzable dimensions. By dynamically synthesizing datasets incorporating up to 20 application-oriented constraints, MOSAIC enables fine-grained, disentangled evaluation. Systematic ablation studies, combined with analyses of constraint composition and positional sensitivity across five mainstream models, reveal non-uniform response patterns dependent on constraint type, count, and placement. The study identifies primacy and recency biases alongside model-specific vulnerabilities, offering critical diagnostic insights to guide the development of more reliable language models.

This study investigates whether small-scale language models adhere to user instructions when those instructions conflict with their task capabilities—such as selecting incorrect answers or generating opposite sentiment—and reveals a decoupling between task proficiency and instruction following. To this end, the authors propose a cross-task evaluation paradigm for conflicting instructions and introduce the Instruction Following Failure Rate (IFFR) metric. Systematic experiments on the Qwen model series demonstrate that while smaller models retain task accuracy, they consistently disregard conflicting instructions, whereas larger models exhibit significantly stronger instruction-following behavior. This work provides the first quantitative evidence that task capability does not equate to controllable behavior, offering a novel perspective and methodology for evaluating model controllability.

This study addresses the instability of large language models in adhering to user instructions when those instructions conflict with patterns established in prior dialogue history. It systematically uncovers, for the first time, the tension between instruction following and pattern completion by evaluating 13 models across 16 instruction types over up to 50 conversational turns using adversarial dialogues. The findings reveal dramatic fluctuations in instruction-following rates (ranging from 1% to 99%), uncorrelated with standard capability benchmarks. Output diversity—particularly in multi-token generation—significantly enhances robustness. Chain-of-thought reasoning frequently exhibits inconsistencies between internal reasoning and final outputs. Although models can predict their own behavior with 83.5% accuracy, they consistently underestimate their resistance to adversarial prompting.