A Status Quo Investigation of Large Language Models towards Cost-Effective CFD Automation with OpenFOAMGPT: ChatGPT vs. Qwen vs. Deepseek

2025-04-02

AI Summary

This study systematically evaluates the effectiveness, stability, and cost-efficiency of ChatGPT, Qwen, and DeepSeek—integrated within the OpenFOAMGPT framework—for automating computational fluid dynamics (CFD) workflows, focusing on boundary condition specification, turbulence model configuration, and solver generation. Method: We propose a novel large language model (LLM) architecture combining instruction fine-tuning and tool augmentation, supporting both zero-shot and expert-guided prompting, and deployable locally (e.g., QwQ-32B) or in the cloud. Contribution/Results: We first empirically reveal the fundamental limitations of zero-shot prompting for complex CFD setups: all models exhibit >40% error rates in critical parameter generation, precluding unsupervised expert replacement. Qwen and DeepSeek achieve performance comparable to ChatGPT on medium-complexity tasks while reducing inference costs substantially. Our work establishes the first empirical benchmark and practical deployment roadmap for industrial-grade LLM-augmented CFD automation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We evaluated the performance of OpenFOAMGPT incorporating multiple large-language models. Some of the present models efficiently manage different CFD tasks such as adjusting boundary conditions, turbulence models, and solver configurations, although their token cost and stability vary. Locally deployed smaller models like QwQ-32B struggled with generating valid solver files for complex processes. Zero-shot prompting commonly failed in simulations with intricate settings, even for large models. Challenges with boundary conditions and solver keywords stress the requirement for expert supervision, indicating that further development is needed to fully automate specialized CFD simulations.

Problem

• Evaluating LLM performance in automating CFD tasks with OpenFOAMGPT
• Assessing token cost and stability variations across different LLM models
• Identifying challenges in zero-shot prompting for complex CFD simulations

Innovation

• Multiple large-language models for CFD tasks
• Local smaller models struggle with complex files
• Zero-shot prompting fails in intricate simulations