StructureClaw: Traceable LLM Agents and an Executable Benchmark for Structural Engineering Workflows

2026-07-16

AI Summary

Current evaluations of large language models predominantly focus on question answering or script generation, which struggle to verify complete and executable chains of evidence in structural engineering and often overlook issues of incomplete workflows or internal inconsistencies. This work proposes StructureClaw-Bench—the first executable benchmark encompassing the full structural engineering workflow—and introduces an artifact-centric agent framework that integrates typed tool invocation, shared artifact state, a local structural analysis engine, and multimodal reconstruction techniques to support traceable and executable pipelines. Leveraging a rigorous success criterion based on artifacts and execution assertions, the approach achieves an 88.6% end-to-end success rate across 50 standard cases, substantially outperforming baseline methods (56.8%) and uncovering two critical challenges: invalid numerical handling and consistent reconstruction of structural models.

Abstract

Addressing a structural-engineering request requires more than a single answer; it requires a chain of interdependent artifacts: interpreted requirements, a computable model, validation records, solver outputs, code-check records, and a final report. Evaluations centered on question answering or script generation rarely verify this complete evidence chain and may therefore reward fluent outputs even when the underlying engineering workflow is incomplete, internally inconsistent, or non-executable. To address this limitation, we present StructureClaw, an artifact-centered workbench in which LLM agents operate through governed engineering skills, typed tools, shared artifact state, and local analysis backends. We also introduce StructureClaw-Bench, an executable benchmark of 150 controlled scenarios spanning standard workflow execution, interactive robustness, and multimodal structural-model reconstruction. A scenario succeeds only when all required artifact- and execution-level assertions pass in a single run. Across ten agent-model configurations, each evaluated on the same 50 standard cases, the average Success Rate rises from 56.8% with the generic-skill baseline to 88.6% with the full automatic workflow. The interactive and multimodal evaluations identify two prominent remaining challenges: safe handling of invalid numerical inputs and fixture-consistent reconstruction of structural models. These findings show that artifact-centered evaluation can expose workflow-level failures that are difficult to identify from final responses alone, providing a more rigorous basis for evaluating and improving structural-engineering agents. The code and benchmark are available at https://github.com/structureclaw/structureclaw.

Problem

• structural engineering
• LLM agents
• workflow evaluation
• artifact chain
• executable benchmark

Innovation

• artifact-centered evaluation
• executable benchmark
• LLM agents
• structural engineering workflows
• typed tools