Evaluating Large Language Models for Real-World Engineering Tasks

2025-05-12

AI Summary

Existing evaluations of large language models’ (LLMs) engineering capabilities rely heavily on simplified exam-style questions and ad hoc scenarios, failing to reflect the complexity and requirements of real-world engineering tasks. Method: We introduce the first production-oriented engineering benchmark—comprising 100+ authentic industrial problems—spanning core competencies including product design, fault diagnosis, and state prediction. We further propose a multidimensional engineering capability evaluation framework that systematically assesses state-of-the-art models (GPT-4, Claude, Llama3, and Qwen) across abstraction reasoning, formal modeling, and context-sensitive engineering logic. Contribution/Results: Empirical results reveal that while models achieve moderate performance on temporal and structural reasoning, their accuracy on critical engineering capabilities remains below 35%, exposing substantial bottlenecks. This work establishes a rigorous, empirically grounded benchmark to characterize and advance LLMs’ engineering proficiency in industrial settings.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large Language Models (LLMs) are transformative not only for daily activities but also for engineering tasks. However, current evaluations of LLMs in engineering exhibit two critical shortcomings: (i) the reliance on simplified use cases, often adapted from examination materials where correctness is easily verifiable, and (ii) the use of ad hoc scenarios that insufficiently capture critical engineering competencies. Consequently, the assessment of LLMs on complex, real-world engineering problems remains largely unexplored. This paper addresses this gap by introducing a curated database comprising over 100 questions derived from authentic, production-oriented engineering scenarios, systematically designed to cover core competencies such as product design, prognosis, and diagnosis. Using this dataset, we evaluate four state-of-the-art LLMs, including both cloud-based and locally hosted instances, to systematically investigate their performance on complex engineering tasks. Our results show that LLMs demonstrate strengths in basic temporal and structural reasoning but struggle significantly with abstract reasoning, formal modeling, and context-sensitive engineering logic.

Problem

• Evaluating LLMs on complex real-world engineering tasks
• Addressing gaps in current LLM engineering assessments
• Assessing LLM performance in core engineering competencies

Innovation

• Curated database of 100+ real-world engineering questions
• Evaluation of four state-of-the-art LLMs
• Systematic coverage of core engineering competencies