CircuitSense: A Hierarchical Circuit System Benchmark Bridging Visual Comprehension and Symbolic Reasoning in Engineering Design Process

2025-09-26

AI Summary

Current multimodal large language models (MLLMs) lack integrated visual understanding and mathematical reasoning capabilities essential for engineering design tasks. Method: We introduce CirBench, the first hierarchical benchmark tailored to circuit design workflows, comprising 8,006+ multi-granularity circuit diagrams—from component-level schematics to system-level block diagrams—and propose a grid-based hierarchical synthesis pipeline with automated symbolic equation annotation, enabling end-to-end evaluation from technical drawings to symbolic equations. Contribution/Results: Experiments reveal that state-of-the-art MLLMs achieve >85% accuracy on visual perception tasks but fall below 19% on symbolic derivation and analytical reasoning—exposing a critical “vision–mathematics” reasoning gap. Our findings establish symbolic reasoning proficiency as a core metric for engineering intelligence, providing a novel, domain-specific benchmark and evaluation paradigm to advance trustworthy MLLM deployment in scientific and engineering applications.

Technology Category

• Computer Vision: Visual Reasoning & Symbolic Representations
• Machine Learning: Probabilistic Circuits and Graphical Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Engineering design operates through hierarchical abstraction from system specifications to component implementations, requiring visual understanding coupled with mathematical reasoning at each level. While Multi-modal Large Language Models (MLLMs) excel at natural image tasks, their ability to extract mathematical models from technical diagrams remains unexplored. We present extbf{CircuitSense}, a comprehensive benchmark evaluating circuit understanding across this hierarchy through 8,006+ problems spanning component-level schematics to system-level block diagrams. Our benchmark uniquely examines the complete engineering workflow: Perception, Analysis, and Design, with a particular emphasis on the critical but underexplored capability of deriving symbolic equations from visual inputs. We introduce a hierarchical synthetic generation pipeline consisting of a grid-based schematic generator and a block diagram generator with auto-derived symbolic equation labels. Comprehensive evaluation of six state-of-the-art MLLMs, including both closed-source and open-source models, reveals fundamental limitations in visual-to-mathematical reasoning. Closed-source models achieve over 85% accuracy on perception tasks involving component recognition and topology identification, yet their performance on symbolic derivation and analytical reasoning falls below 19%, exposing a critical gap between visual parsing and symbolic reasoning. Models with stronger symbolic reasoning capabilities consistently achieve higher design task accuracy, confirming the fundamental role of mathematical understanding in circuit synthesis and establishing symbolic reasoning as the key metric for engineering competence.

Problem

• Evaluating MLLMs' ability to understand hierarchical circuit diagrams
• Bridging visual comprehension with symbolic mathematical reasoning
• Assessing derivation of equations from technical schematics and diagrams

Innovation

• Hierarchical synthetic generation pipeline for circuits
• Grid-based schematic and block diagram generators
• Auto-derived symbolic equation labels for training