🤖 AI Summary
This study addresses the limitations of existing analog circuit benchmarks in evaluating model generalization and practical usability by constructing an end-to-end benchmark comprising 273 designs derived from Tiny Tapeout open-source fabrication data. Leveraging real manufacturing data for the first time, this work establishes a containerized environment that enables precise source-code-to-SPICE mapping and training cutoff analysis. Furthermore, it incorporates automated export pipelines and graph isomorphism matching algorithms to cover schematic-to-netlist transcription and device sizing optimization tasks. Experimental results demonstrate that the best-performing model achieves a graph isomorphism accuracy of 56.1% and a sizing optimization score of 91.2. These findings reveal significant discrepancies between visual transcription and design capabilities, highlighting critical strategic bottlenecks in current models.
📝 Abstract
Existing analog integrated circuit design benchmarks make two questions hard to answer: whether a model has learned transferable circuit skills rather than recalled familiar examples, and whether its output works under defined process and test conditions. We introduce a benchmark built from human-designed, open-source circuits associated with Tiny Tapeout manufacturing shuttles. The collection contains 273 topologically distinct top-level designs. Every source is retrieved at the revision recorded for its shuttle submission and processed in a fixed containerized environment. The pipeline exports each eligible schematic image and its SPICE netlist from the same source file, giving transcription an exact structural reference. Commit dates support model-specific training-cutoff analysis, while author testbenches provide the simulation context for sizing. The benchmark evaluates schematic-to-netlist transcription and device sizing. Across seven models on a fixed set of 66 transcription tasks, the strongest model reaches 56.1% exact graph isomorphism, and six of seven models drop sharply from the small to the medium tier. For one frontier model, removing author-chosen labels reduces exact matches while preserving aggregate structural F1, suggesting that labels can aid connectivity tracing. On the 17 sizing tasks, the leading model converges on all 17 proposals and reaches 91.2 out of 100 against the human reference, while the two newest Claude models refuse 4 and 11 of the same prompts they transcribe without objection; a proposal without sizes scores zero. The two tasks produce different model rankings, exposing distinct visual and design capabilities and, in one family, a policy rather than capability limit.