OWLViz: An Open-World Benchmark for Visual Question Answering

2025-03-04

AI Summary

Open-world visual question answering (OW-VQA) demands joint visual understanding, web retrieval, and dynamic tool invocation to answer precise cross-modal questions—yet no benchmark systematically evaluates multi-step tool orchestration and structured reasoning. Method: We introduce OWLViz, the first end-to-end multimodal benchmark for OW-VQA, built on real-world web pages with human-verified query-answer pairs. It explicitly requires models to select, sequence, and reason over heterogeneous tools—including search APIs and vision-language models—under open-world constraints. Contribution/Results: OWLViz is the first to formally define and evaluate dynamic tool coordination and long-horizon reasoning, exposing fundamental limitations of current VLMs in tool selection, planning depth, and reasoning-chain modeling. Human accuracy is 69.2%, while the best VLM (Gemini 2.0) achieves only 26.6%; embodied VLM agents perform worse. Results underscore the critical need for structured reasoning architectures and robust tool orchestration frameworks in open-world multimodal AI.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Question Answering

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web search models and ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

We present a challenging benchmark for the Open WorLd VISual question answering (OWLViz) task. OWLViz presents concise, unambiguous queries that require integrating multiple capabilities, including visual understanding, web exploration, and specialized tool usage. While humans achieve 69.2% accuracy on these intuitive tasks, even state-of-the-art VLMs struggle, with the best model, Gemini 2.0, achieving only 26.6% accuracy. Current agentic VLMs, which rely on limited vision and vision-language models as tools, perform even worse. This performance gap reveals significant limitations in multimodal systems' ability to select appropriate tools and execute complex reasoning sequences, establishing new directions for advancing practical AI research.

Problem

• Challenges in Open-World Visual Question Answering tasks.
• Integration of visual understanding, web exploration, and tool usage.
• Performance gap between humans and state-of-the-art VLMs.

Innovation

• Open-world benchmark for visual question answering
• Integrates visual understanding and web exploration
• Highlights limitations in multimodal AI systems