Enginuity: A Dataset and Benchmark for Vision-Language Understanding of Engineering Diagrams

2026-06-02

AI Summary

Existing vision-language model evaluation benchmarks inadequately address the distinctive characteristics of engineering drawings, such as dense layouts, specialized symbols, and cross-references between text and graphics. This work introduces the first open dataset and benchmark specifically designed for complex engineering drawings, proposing two tasks: structured parts-list extraction and free-form visual question answering. The study employs zero-shot learning, chain-of-thought prompting, and LLM-as-judge calibration for systematic evaluation. Experimental results show that state-of-the-art models achieve part-recognition recall rates of 0.61–0.87 but exhibit poor performance in descriptive token F1 scores (0.03–0.18) and generally underperform on visual question answering, revealing significant deficiencies in technical semantic understanding and factual reasoning. Furthermore, the findings demonstrate that conventional overlap-based metrics substantially underestimate models’ capacity for technical description.

Abstract

Engineering diagrams pose a distinct challenge for vision-language models: unlike natural images or general documents, they encode information through dense spatial layouts, domain-specific symbols, and cross-references between visual callouts and structured parts tables. Despite their centrality to service, repair, and design workflows, there is no public benchmark for measuring VLM capabilities in this domain; existing datasets primarily focus on flowcharts, scientific figures, or business documents. To address this gap, we introduce Enginuity, the first open dataset and benchmark for evaluating VLMs on complex engineering diagrams. We define two tasks over a corpus of U.S. military service and repair manuals: structured parts-table extraction (Task 1) and free-form visual diagram question answering (VQA)(Task 2) for benchmarking. We evaluate four frontier VLMs (GPT-5.2 Chat, Claude Opus 4.7, Gemma 4, Qwen3-VL-32B-Instruct) under zero-shot and chain-of-thought prompting. On Task 1, models reach Recall@all of 0.61-0.87 but Token F1pen of only 0.03-0.18, exposing a systematic gap between part identification and description fidelity. Task 2 reveals a consistent factual-reasoning gap across all models. A supporting analysis shows that token-overlap metrics under-report model capability on technical descriptions by 2-6x relative to semantic similarity, motivating LLM-as-judge calibration for domain-specific evaluation. We release the dataset, annotations, evaluation harness, and per-sample model outputs to support a reproducible study of VLM capability on engineering content.

Problem

• engineering diagrams
• vision-language models
• benchmark
• dataset
• visual question answering

Innovation

• engineering diagrams
• vision-language models
• structured extraction
• visual question answering
• LLM-as-judge