Tailoring Chatbots for Higher Education: Some Insights and Experiences

2024-08-26

arXiv.org

AI Summary

General-purpose large language models (LLMs) suffer from domain knowledge gaps and low response accuracy in higher education–specific tasks. To address the methodological gap in adapting LLMs to education verticals, this paper proposes a lightweight, university-scenario–oriented LLM customization framework. The framework integrates discipline-specific knowledge injection, role-aware prompt engineering, and a localized human-in-the-loop evaluation mechanism. Leveraging open-source base models, it incorporates educational corpus construction, knowledge-augmented fine-tuning, and multi-dimensional evaluation techniques. Deployed as a teaching-assistant chatbot at ETH Zurich, the system achieves a 32% improvement in FAQ answer accuracy and significantly enhanced contextual relevance. Empirical results validate the framework’s feasibility, effectiveness, and cross-task reusability for domain-adaptive LLM deployment in higher education.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The general availability of powerful Large Language Models had a powerful impact on higher education, yet general models may not always be useful for the associated specialized tasks. When using these models, oftentimes the need for particular domain knowledge becomes quickly apparent, and the desire for customized bots arises. Customization holds the promise of leading to more accurate and contextually relevant responses, enhancing the educational experience. The purpose of this short technical experience report is to describe what"customizing"Large Language Models means in practical terms for higher education institutions. This report thus relates insights and experiences from one particular technical university in Switzerland, ETH Zurich.

Problem

• Customizing chatbots for specialized higher education tasks
• Addressing domain knowledge gaps in general-purpose language models
• Enhancing educational experience with contextually relevant responses

Innovation

• Customized chatbots for specialized higher education tasks
• Domain knowledge integration to improve response accuracy
• Practical implementation of tailored large language models