A Lightweight Multi-Expert Generative Language Model System for Engineering Information and Knowledge Extraction

2025-05-27

AI Summary

To address the pervasive hallucination issue in engineering-domain text generation and the high computational cost and low resource efficiency of existing fine-tuning methods, this paper proposes SLG—a lightweight multi-expert generative language model system. Methodologically, SLG introduces the novel “Small Language Graph” (SLG) architecture: it organizes multiple finely tuned small language models (SLMs) as expert nodes in a graph structure, employs graph neural networks for task decomposition and collaborative multi-expert reasoning, and integrates engineering-domain corpus refinement with structured prompt optimization. Experimental results demonstrate that SLG achieves three times higher Exact Match accuracy than conventional fine-tuning approaches, accelerates fine-tuning by 1.7×, substantially mitigates hallucinations, and enables deployment on edge devices—thereby achieving a favorable trade-off between high reliability and low resource consumption.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Prompt Engineering / Prompting
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Despite recent advancements in domain adaptation techniques for large language models, these methods remain computationally intensive, and the resulting models can still exhibit hallucination issues. Most existing adaptation methods do not prioritize reducing the computational resources required for fine-tuning and inference of language models. Hallucination issues have gradually decreased with each new model release. However, they remain prevalent in engineering contexts, where generating well-structured text with minimal errors and inconsistencies is critical. This work introduces a novel approach called the Small Language Graph (SLG), which is a lightweight adaptation solution designed to address the two key challenges outlined above. The system is structured in the form of a graph, where each node represents a lightweight expert - a small language model fine-tuned on specific and concise texts. The results of this study have shown that SLG was able to surpass conventional fine-tuning methods on the Exact Match metric by 3 times. Additionally, the fine-tuning process was 1.7 times faster compared to that of a larger stand-alone language model. These findings introduce a potential for small to medium-sized engineering companies to confidently use generative AI technologies, such as LLMs, without the necessity to invest in expensive computational resources. Also, the graph architecture and the small size of expert nodes offer a possible opportunity for distributed AI systems, thus potentially diverting the global need for expensive centralized compute clusters.

Problem

• Reduce computational resources for fine-tuning language models
• Minimize hallucination issues in engineering contexts
• Enable small companies to use generative AI affordably

Innovation

• Lightweight Multi-Expert Graph System
• Small Language Model Fine-Tuning
• Distributed AI Architecture