Which Constraints Are Missing? Ask the Verifier: Graded Rewards for Constraint-Following Music Generation

2026-09-20

AI Summary

该研究通过构建MusicConstraintBench和提出MusicRLVR方法，解决了音乐生成中多约束同时满足的问题，提高了模型在复杂约束下的表现。

Abstract

Constraint-following music generation asks a score to satisfy several user-specified properties at once, each checkable programmatically (key, meter, length, range, final note, rhythm, motion and form), yet no existing benchmark isolates this capability. We construct MusicConstraintBench, 2,180 items over eight constraint families, on which current models fail once a few constraints are combined. The natural remedy is reinforcement learning with these verifiers as reward, yet we observe that a reward paid only when every property holds leaves most training groups without a learning signal: over the first 50 updates, 0.550 of rollout groups score identically and receive no gradient, even though a failing score typically misses only one requested property. Under the joint criterion, rollouts for a prompt tend to fail together, so a binary reward cannot separate a nearly correct score from a malformed one. We therefore introduce MusicRLVR, which pays graded per-property credit behind a hard validation gate that rejects malformed outputs, plus a joint-satisfaction bonus, requiring no human annotation, learned reward model, or music-domain fine-tuning. On MusicConstraintBench, MusicRLVR lifts Qwen3-4B-Instruct from 0.160 to 0.807 on mixed constraints and leads every zero-shot baseline including Llama-3.1-70B at 0.380. It also generalises to property combinations unseen in training and to out-of-range parameter values, showing that verifiable rewards need not presuppose a target output.

Problem

• constraint-following
• music generation
• reinforcement learning
• reward signal
• multiple constraints

Innovation

• Graded Rewards
• Constraint-Following Music Generation
• Reinforcement Learning
• MusicRLVR