PRISM: Priority-aware Rubric Internalization via Structured Multimodal Data Synthesis

2026-08-05

AI Summary

Real-world multimodal instructions often entail multiple requirements with varying priorities, yet existing models lack the ability to understand and execute structured scoring rules. This work proposes PRISM, a novel framework that formulates rule adherence as an execution task. PRISM synthesizes training data through four stages: role–task pair generation, prefix-guided rule set construction, quality filtering, and structured validation trajectories. It further introduces Loose/Strict evaluation metrics that eliminate the need for a judge model at inference time. Using only 10,000 synthetic samples, Qwen3-VL-4B achieves a substantial improvement in Strict accuracy on PRISM-Eval—from 9.5% to 30.1%—while demonstrating strong generalization across four open-source multimodal large language models and maintaining competitive performance on standard benchmarks.

Abstract

Real-world multimodal instructions often bundle multiple requirements with unequal importance, yet most multimodal training data still reduce instruction following to answering one self-contained question. We study this gap through \textbf{rubric comprehension}, which casts the model not as a generator measured against rubrics but as an \textbf{executor} that follows them: given an image and a typed, prioritized rubric, the model must verify each rule before producing an overall judgment. To support this setting, we propose \textbf{PRISM}, a four-stage data synthesis framework that produces persona--task pairs, prefix-guided rule sets, quality-filtered rubrics, and structured verification traces. We further introduce \textbf{PRISM-Eval}, whose Loose and Strict metrics use deterministic matching against fixed labels and therefore require no inference-time judge model. With only 10K synthesized samples, PRISM lifts Qwen3-VL-4B from 9.5\% to 30.1\% Strict accuracy on PRISM-Eval while preserving average performance on general benchmarks, and the gains transfer to four additional open-source MLLMs across dense and MoE architectures, suggesting that structured rubric supervision is a scalable path toward multi-rule, priority-aware multimodal instruction following.

Problem

• multimodal instruction following
• rubric comprehension
• priority-aware reasoning
• structured multimodal data
• multi-rule execution

Innovation

• rubric comprehension
• structured multimodal synthesis
• priority-aware instruction following
• verification traces
• PRISM-Eval