Querying Large Automotive Software Models: Agentic vs. Direct LLM Approaches

2025-06-16

AI Summary

Natural language querying over ultra-large-scale Ecore software models in the automotive domain poses significant challenges, particularly under strict privacy, intellectual property, and regulatory compliance constraints. Method: This paper proposes a lightweight, local large language model (LLM)-driven agent framework that replaces computationally expensive direct prompting with a multi-step agent workflow. The framework integrates filesystem interaction tools and Ecore metamodel parsing capabilities to enable secure, on-premises model analysis. Contribution/Results: Experiments demonstrate that the approach achieves accuracy comparable to direct prompting while reducing token consumption by over an order of magnitude. It represents the first empirical validation of small, local LLM agents for automotive software modeling—establishing their feasibility and superiority in privacy-sensitive, compliance-critical industrial settings. The framework provides the only viable lightweight solution for confidential, regulatory-compliant model understanding in automotive embedded systems.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Cognitive Modeling & Cognitive Systems: Agent ArchitecturesNatural Language Processing: (Large) Language Models

Application Category

Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactionsUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendationEconomics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large language models (LLMs) offer new opportunities for interacting with complex software artifacts, such as software models, through natural language. They present especially promising benefits for large software models that are difficult to grasp in their entirety, making traditional interaction and analysis approaches challenging. This paper investigates two approaches for leveraging LLMs to answer questions over software models: direct prompting, where the whole software model is provided in the context, and an agentic approach combining LLM-based agents with general-purpose file access tools. We evaluate these approaches using an Ecore metamodel designed for timing analysis and software optimization in automotive and embedded domains. Our findings show that while the agentic approach achieves accuracy comparable to direct prompting, it is significantly more efficient in terms of token usage. This efficiency makes the agentic approach particularly suitable for the automotive industry, where the large size of software models makes direct prompting infeasible, establishing LLM agents as not just a practical alternative but the only viable solution. Notably, the evaluation was conducted using small LLMs, which are more feasible to be executed locally - an essential advantage for meeting strict requirements around privacy, intellectual property protection, and regulatory compliance. Future work will investigate software models in diverse formats, explore more complex agent architectures, and extend agentic workflows to support not only querying but also modification of software models.

Problem

• Comparing agentic vs direct LLM approaches for querying large software models
• Evaluating efficiency and accuracy of LLM methods in automotive software analysis
• Addressing privacy and scalability challenges in automotive model querying

Innovation

• Agentic LLM approach with file access tools
• Efficient token usage for large models
• Local execution with small LLMs