MUSE: A Unified Agentic Harness for MLLMs

2026-06-01

AI Summary

This work addresses the underperformance of multimodal large language models (MLLMs) on seemingly human-easy tasks such as visual spatial reasoning, attributing this limitation primarily to their execution frameworks rather than the models themselves. To overcome this, the authors propose MUSE—a unified, structured execution framework that requires no model retraining. MUSE enhances the reasoning capabilities of any off-the-shelf MLM by modularly integrating task representation, visual perception, tool invocation, structured parsing, deterministic verification, and a novel verifier-guided repair mechanism. Notably, MUSE introduces verifier-guided repair for the first time, substantially boosting performance without altering the underlying model and thereby charting a new path beyond model-centric optimization. Experimental results demonstrate consistent improvements over strong baselines across multiple vision and multimodal benchmarks, with particularly pronounced gains on challenging samples.

Abstract

Despite rapid progress, multimodal large language models (MLLMs) still fail on tasks that humans solve effortlessly, such as navigating a grid maze from a screenshot or selecting the correct puzzle piece. Rather than retraining the model, we ask a complementary question: how much capability can be elicited from a frozen MLLM purely by improving the execution scaffold around it? We introduce MUSE, a multimodal unified structured execution harness that wraps any off-the-shelf MLLM with composable modules for task representation, visual processing, perception tool use, structured parsing, deterministic verification, and verifier-guided repair, without any model retraining. We evaluate MUSE across diverse benchmarks spanning visual spatial planning, visual perception, multimodal reasoning, and fine-grained visual discrimination, using multiple state-of-the-art MLLMs. MUSE delivers consistent gains over the bare model in all settings, with the largest jumps on challenging instances. Further analysis reveals that many MLLM failures arise from harness-level shortcomings rather than fundamental model deficits, and can be addressed through verifier-guided repair without touching the model. These findings highlight the agentic multimodal harness as a critical yet underexplored design dimension, offering an orthogonal avenue for improving MLLMs beyond model-centric optimization.

Problem

• multimodal large language models
• execution scaffold
• visual reasoning
• model failures
• agentic harness

Innovation

• multimodal large language models
• structured execution harness
• verifier-guided repair
• agentic framework
• zero-retraining enhancement