design multimodal benchmarks

Existing multimodal evaluation benchmarks inadequately reflect real-world, heterogeneous daily usage scenarios and lack systematic assessment across diverse tasks and output formats. Method: We introduce the first fine-grained, real-scenario-oriented multimodal benchmark—comprising 505 practical scenarios and 8,000+ samples—supporting 16 input/output modalities and 40+ output formats (e.g., numbers, code, JSON, free-form text). We propose a four-dimensional capability reporting framework—“Application–Input–Output–Skill”—replacing monolithic multiple-choice evaluation with task-driven, format-aware, interpretable assessment. The benchmark integrates expert crowdsourced scenario sampling, 40+ customized automated metrics, multi-format parsers, and interactive visualization tools. Contribution/Results: Comprehensive evaluation of state-of-the-art vision-language models reveals, for the first time, their fine-grained capability boundaries and long-tail deficiencies across modality combinations and task types.

Current multimodal fusion evaluation is hindered by small-scale, narrow-domain, task-specific, and inconsistently standardized benchmarks, leading to poor model generalizability and incomparable results. To address this, we propose MMBench—the first large-scale, domain-adaptive multimodal fusion benchmark—integrating over 30 datasets, 15 modalities, and 20 predictive tasks across critical domains including healthcare, remote sensing, and industrial inspection. We design a unified cross-domain evaluation framework and an open-source automated pipeline supporting early-, late-, and hybrid-fusion paradigms. Our framework incorporates standardized preprocessing, cross-modal alignment, and domain-adaptation mechanisms. Extensive experiments establish multiple new state-of-the-art baselines, significantly improving model generalizability and reproducibility. MMBench provides a rigorous, open, and extensible evaluation infrastructure for advancing multimodal fusion research.

Existing benchmarks lack systematic evaluation of multimodal tool orchestration under model-context protocols, especially for complex multi-hop, multithreaded workflows involving visual grounding, cross-tool dependencies, and persistent multi-step intermediate states. Method: We introduce MToolBench—the first benchmark dedicated to multimodal tool usage—comprising 231 real-world tools deployed across 28 servers, enabling end-to-end multimodal workflow evaluation. It features a novel similarity-driven trajectory alignment method: tool signatures are embedded via sentence encoders; similarity-based bucketing and the Hungarian algorithm jointly enable auditable, one-to-one tool-call matching, decoupling semantic fidelity from procedural consistency. Execution traces are validated by an integrated executor and a four-model adjudication panel. Contribution/Results: Experiments reveal significant bottlenecks in current multimodal LMs regarding parameter-level accuracy and structural coherence, underscoring the urgent need for joint image–text–tool graph reasoning.

Multimodal large language models (MLLMs) benchmarks suffer from pervasive non-visual shortcut learning—models achieve high scores by exploiting textual biases, linguistic priors, or superficial statistical patterns, severely compromising the validity of visual understanding evaluation. Method: We propose a “test-set stress testing” and “iterative bias pruning” framework that leverages LLMs to actively detect and quantify textual biases in benchmarks. Using k-fold cross-validation, we fine-tune an LLM and integrate it with random forests and handcrafted features to score and prune biased samples. Contribution/Results: Our method systematically identifies and eliminates non-visually solvable instances across four mainstream benchmarks, yielding the debiased benchmark VSI-Bench-Debiased. It exhibits significantly reduced non-visual solvability, widened performance gaps on visually blind tasks, and robustly advances a vision-centric, reliable paradigm for multimodal evaluation.

Existing multimodal LLM (MLM) evaluation benchmarks inadequately assess fine-grained capabilities—such as object recognition, attribute understanding, and spatiotemporal or relational reasoning—in realistic user-facing scenarios. To address this, we propose the first user-query-driven dynamic benchmark generation paradigm. Our approach integrates taxonomy-aware multimodal asset classification, procedural task synthesis, constraint-aware query optimization, and cross-modal automatic alignment annotation, enabling scalable construction of a hierarchical multimodal asset taxonomy and a million-scale controllable QA-pair generation algorithm. The system supports three modalities—images, videos, and 3D objects—and generates 750 million high-quality QA pairs. Empirically, it reveals, for the first time, systematic structural deficiencies of open-source MLMs in spatiotemporal reasoning, precisely characterizing their capability boundaries. This facilitates scenario-driven model selection and targeted optimization.

This work addresses the current lack of systematic and reusable methodologies for evaluating multimodal user interface toolkits across critical dimensions such as modality support, development efficiency, and experimental integration. The paper introduces the first structured benchmarking framework, enabling a comprehensive comparison of representative toolkits—including Geno, MSP, ReactGenie, WAMI, and EmoSync—along three axes: modality coverage and interaction abstraction, developer experience and workflow, and support for experimentation and system integration. Through document analysis, technical comparisons, and planned developer studies, the authors construct a reusable benchmark template that establishes a standardized foundation for future evaluation, selection, and empirical research on multimodal UI toolkits.

This work addresses the lack of evaluation frameworks for assessing multimodal large language models’ ability to jointly reason about data semantics, view coordination, and interaction logic in collaborative multi-view interface construction. The authors propose MV-Bench, the first task-specific benchmark built upon real-world Tableau workbooks, which leverages a structured intermediate representation and an executable web interface generation pipeline. MV-Bench encompasses 92 base interfaces and 1,048 validation instances, and introduces an automated evaluation along three dimensions: visual fidelity, data-binding correctness, and interaction completeness. While the strongest model achieves 75.45% accuracy in layout reproduction, its performance drops significantly in data binding (21.71%) and interaction completeness (11.68%), revealing critical bottlenecks in current models’ capacity for semantic understanding and interactive behavior generation.

This work addresses a critical limitation in conventional large language model (LLM) evaluation, which treats benchmark datasets as homogeneous aggregates and overlooks the heterogeneity among samples in cognitive, linguistic, and task-related attributes. The authors propose a dataset-centric meta-evaluation framework that introduces fine-grained, sample-level annotations across five dimensions: cognitive demand, language quality, task characteristics, contextual dependency, and ethical safety. For the first time, this approach enables multidimensional auditing of widely used benchmarks such as MMLU and ARC. By allowing dynamic subset composition aligned with specific evaluation objectives, the framework uncovers the diversity obscured by aggregate accuracy metrics and establishes a composable evaluation paradigm tailored to targeted capabilities—such as reasoning depth or ethical sensitivity—thereby substantially enhancing the precision and interpretability of LLM assessments.

Existing static multimodal evaluation benchmarks are hindered by temporal degradation, data contamination, and high maintenance costs, limiting their ability to sustainably assess vision-language models. This work proposes a multi-agent-driven, automated dynamic evaluation framework that models benchmark evolution as a task-guided dataset construction process. By integrating structured specifications, feedback-controlled real-time data collection, and verifiable question-answer generation, the framework enables efficient and continuous updates. A novel distribution-consistency update strategy is introduced to preserve cross-version comparability while substantially mitigating data contamination risks. Experiments demonstrate that the approach can generate 5.9K high-quality samples in just 1–2 hours at a cost of approximately \$30, effectively maintaining model ranking stability and semantic consistency while significantly reducing memorization signals.