IFHierBench: Hierarchical Instruction Following for Large Language Models

2026-07-30

AI Summary

Existing instruction-following benchmarks struggle to evaluate large language models’ ability to satisfy hierarchical constraints across nested output structures. This work proposes the first evaluation framework specifically designed for hierarchical constraint adherence, introducing a benchmark dataset comprising 600 prompts that span four levels of constraint-tree depth and 35 distinct constraint types. Each prompt is paired with a deterministic checker enabling multi-granular validation—from entire outputs down to individual nested fields. Experiments across seven mainstream large language models reveal that even the strongest model achieves only slightly above 50% prompt-level accuracy, with performance degrading significantly as constraint depth increases, thereby exposing a critical limitation in current models’ capacity to handle nested constraints.

Abstract

Instruction-following ability is critical for deploying large language models in real-world applications, where downstream components depend on the output satisfying specific constraints. Modern deployments increasingly handle the full task in a single LLM call, with one prompt specifying a layered output whose overall artifact, structural sections, and nested fields must each satisfy concrete constraints. Existing instruction-following benchmarks treat the constraint set as a flat list applied uniformly to the response, so they cannot scope a check to a particular section of the output. We introduce IFHierBench, a hierarchical instruction-following benchmark of 600 prompts stratified across four constraint-tree depths and 35 distinct constraints, each prompt paired with a deterministic checker that verifies satisfaction at every scope. Evaluating seven leading proprietary and open-weight models, we find that even the strongest model only marginally exceeds 50% prompt-level accuracy and that accuracy degrades sharply as constraint depth grows. Reliably following nested constraints remains a substantial gap for current LLMs, motivating future training methods that consider constraint adherence at finer granularity to achieve better instruction-following ability.

Problem

• instruction following
• hierarchical constraints
• large language models
• nested constraints
• benchmarking

Innovation

• hierarchical instruction following
• constraint scoping
• structured output verification
• LLM benchmarking
• nested constraints