Large Language Model for Qualitative Research - A Systematic Mapping Study

2024-11-18

arXiv.org

AI Summary

Traditional qualitative analysis suffers from low efficiency and high subjectivity in large-scale text processing. To address this, this study conducts a systematic mapping study (SMS) to construct, for the first time, a comprehensive knowledge graph of large language model (LLM)-augmented qualitative research, spanning healthcare, education, and social sciences. It systematically characterizes LLM applications across core qualitative tasks—including coding and theme extraction—alongside configuration strategies, methodological foundations, and evaluation metrics. Key contributions include: (1) identifying “human-AI collaboration” as an emerging paradigm; (2) proposing technical pathways to enhance model robustness and mitigate factual hallucinations and contextual limitations; and (3) designing a human-in-the-loop augmentation framework coupled with a multi-dimensional evaluation system. Results demonstrate that LLMs significantly improve analytical efficiency; however, their reliability critically depends on prompt engineering quality and domain-specific adaptation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Qualitative Reasoning
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

The exponential growth of text-based data in domains such as healthcare, education, and social sciences has outpaced the capacity of traditional qualitative analysis methods, which are time-intensive and prone to subjectivity. Large Language Models (LLMs), powered by advanced generative AI, have emerged as transformative tools capable of automating and enhancing qualitative analysis. This study systematically maps the literature on the use of LLMs for qualitative research, exploring their application contexts, configurations, methodologies, and evaluation metrics. Findings reveal that LLMs are utilized across diverse fields, demonstrating the potential to automate processes traditionally requiring extensive human input. However, challenges such as reliance on prompt engineering, occasional inaccuracies, and contextual limitations remain significant barriers. This research highlights opportunities for integrating LLMs with human expertise, improving model robustness, and refining evaluation methodologies. By synthesizing trends and identifying research gaps, this study aims to guide future innovations in the application of LLMs for qualitative analysis.

Problem

• Automating qualitative analysis with LLMs
• Exploring LLM applications in diverse fields
• Addressing LLM challenges in qualitative research

Innovation

• LLMs automate qualitative analysis
• Integrate LLMs with human expertise
• Improve model robustness and evaluation