MMOOC: A Comprehensive Benchmark for Out-of-Context Evaluation in Multimodal Large Language Models

2026-07-29

AI Summary

This work addresses the challenge that multimodal large language models (MLLMs) struggle to distinguish between out-of-context questions—requiring rejection—and in-context shifted questions—amenable to answering—under contextual shift scenarios, a gap inadequately covered by existing benchmarks. To this end, we introduce MMOOC, a large-scale evaluation benchmark that systematically encompasses eight types of contextual shifts, three question formats, and diverse visual settings. Rigorous multi-stage data cleaning and human verification ensure high data quality. We further propose an LLM-as-a-Judge-based metric to assess reasoning correctness. Experimental results reveal a significant imbalance in current models’ ability to appropriately reject or answer such questions, while post-training strategies demonstrate notable improvements in robustness. The benchmark will be publicly released.

Abstract

Multimodal Large Language Models (MLLMs) have achieved strong performance on a wide range of vision-language tasks, but often fail under imperfect or shifted contexts. A reliable MLLM should refuse truly out-of-context (OOC) questions with subject-level context shifts while still answering shifted in-context (Shifted IC) questions with non-subject context shifts. Existing benchmarks mainly target OOC or visually unanswerable questions, but overlook answerable Shifted IC cases and cover limited OOC shifts. To fill this gap, we present MMOOC, a large-scale benchmark for evaluating refusal and robust answering abilities of MLLMs. MMOOC contains over 41K image-question pairs, including answerable Shifted IC cases and unanswerable OOC cases, spanning three question formats, eight shift types and six visual scenarios, with data quality ensured through MLLM-based filtering and human verification. We evaluate model responses using Accuracy and Refusal Rate, and further introduce an LLM-as-a-Judge metric to assess the correctness of model reasoning. Experiments on diverse MLLMs show that current models still struggle to balance answer-ability and refusal under shifted contexts. We further analyze key failure patterns and show that post-training can improve robustness. MMOOC will be made publicly available.

Problem

• out-of-context
• multimodal large language models
• context shift
• refusal ability
• robustness evaluation

Innovation

• out-of-context evaluation
• multimodal large language models
• context shift
• refusal behavior
• robustness benchmark