NoRA: Evaluating Grounded Reasonableness in Visual First-person Normative Action Reasoning

2026-06-03

AI Summary

Existing approaches are limited to textual evaluation or fixed-choice selection, making it difficult to assess an agent’s ability to generate and justify appropriate actions from scratch based on observable facts. To address this gap, this work proposes NoRA—a first-person visual video benchmark that requires models to generate candidate next actions and substantiate them through an explicit fact–reason–action support graph. We introduce, for the first time, an “evidenced plausibility” evaluation framework that shifts normative reasoning from option selection toward verifiable justification, incorporating three metrics: action alignment, fact grounding, and support binding. These are evaluated under direct, deliberative, and structured prompting strategies across multimodal large language models. Experiments on twelve systems reveal that while current models can identify plausible actions and relevant facts, they exhibit significant shortcomings in constructing a complete action space and correctly binding local supports.

Abstract

LLMs and agentic systems are increasingly deployed in social environments, making normative competence critical for safe and appropriate behavior. However, existing approaches either assess normative judgment in text alone or reduce it to choosing among a fixed set of candidate actions. We argue both are insufficient. In practice, agents are never handed a menu of options; they must identify a reasonable action from scratch, grounded in visible facts and supported by inspectable reasons. We introduce NoRA, a visual first-person video benchmark that requires models to generate candidate next actions and justify each through an explicit fact-reason-action support graph. The benchmark comprises 1,420 annotated video clips, including HumanGold-190 and LLMSilver-1230 splits. Each instance is evaluated through action alignment, factual grounding, and support binding, aggregated into a single grounded reasonableness score. We benchmark 12 multimodal systems under direct, deliberate, and structured prompting regimes, finding that current VLMs frequently recover plausible actions and relevant scene facts, but consistently struggle to construct the full reasonable action space and bind selected actions to the correct local support. NoRA makes this gap measurable, shifting the evaluation question from whether a model can pick an action to whether it can justify an appropriate action for the right visible reasons.

Problem

• normative reasoning
• visual grounding
• action justification
• first-person video
• reasonableness evaluation

Innovation

• grounded reasonableness
• normative action reasoning
• visual first-person benchmark
• fact-reason-action graph
• multimodal evaluation