SyRuP: Enhancing System-Prompt Following via Reward-Guided Prediction in LLM Decoding

2026-07-27

AI Summary

This work addresses the challenge that large language models struggle to effectively follow complex or compositional system prompts using only in-context learning. To overcome this limitation, the authors propose SyRuP—a decoding-time framework that requires no fine-tuning and introduces the first system-prompt-based cross-attention reward head to enable fine-grained, token-level compliance scoring. SyRuP enhances candidate token reranking by integrating preference-pair training, token-level reward signals, and an optional contrastive logit shift. Experimental results demonstrate that SyRuP significantly outperforms existing methods across multiple system prompt adherence benchmarks, achieving strong performance with only moderate inference overhead.

Abstract

Large Language Models (LLMs) are increasingly controlled through system prompts that specify roles, styles, formats, and safety requirements. However, models follow these prompts only implicitly through in-context learning, which can be insufficient for complex or compositional prompts. Existing approaches often require model tuning or response-level reranking, limiting their practicality for lightweight inference-time control. We introduce SyRuP, a decoding-time framework for improving system-prompt adherence while keeping the base LM frozen. SyRuP trains a cross-attention reward head from system-prompt-conditioned preference pairs, treating the system prompt as a separate memory to produce token-level adherence scores. At inference, SyRuP reranks the base LM's top-k candidates by combining base logits with the learned reward signal and an optional contrastive signal capturing system-induced logit shifts. Experiments on system-prompt following benchmarks show that SyRuP consistently outperforms prompting and decoding-time baselines with moderate inference overhead. These results suggest that explicit token-level guidance is an effective and practical mechanism for reliable system-prompt following.

Problem

• system-prompt following
• Large Language Models
• prompt adherence
• decoding-time control
• token-level guidance

Innovation

• system-prompt following
• reward-guided decoding
• token-level reward
• frozen LLM
• cross-attention reward head