On the Use of Large Language Models for Qualitative Synthesis

2025-10-18

AI Summary

This study addresses methodological risks arising from the application of large language models (LLMs) in the qualitative synthesis (QS) phase of systematic reviews—particularly bias amplification and diminished result credibility. Adopting a novel collaborative reflexive research paradigm, it integrates LLM technical mechanism analysis, iterative empirical trials, and methodological critique frameworks to systematically evaluate LLMs’ text summarization capabilities and practical limitations. Findings reveal that inconsistent methodological reporting frequently leads to LLM misuse, while intrinsic structural properties—including training data biases and opaque reasoning processes—significantly undermine the reproducibility and interpretability of synthesized outputs. Based on these insights, the study proposes three foundational principles: human-led oversight, model-assisted execution, and process transparency—highlighting the critical role of embedded supervision and auditable operational logging in safeguarding QS rigor. This work delivers the first reflexively grounded risk identification framework and governance pathway for integrating LLMs into evidence-based research.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Qualitative Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large language models (LLMs) show promise for supporting systematic reviews (SR), even complex tasks such as qualitative synthesis (QS). However, applying them to a stage that is unevenly reported and variably conducted carries important risks: misuse can amplify existing weaknesses and erode confidence in the SR findings. To examine the challenges of using LLMs for QS, we conducted a collaborative autoethnography involving two trials. We evaluated each trial for methodological rigor and practical usefulness, and interpreted the results through a technical lens informed by how LLMs are built and their current limitations.

Problem

• Applying LLMs to qualitative synthesis in systematic reviews
• Examining risks of misuse that amplify existing weaknesses
• Evaluating methodological rigor and practical usefulness of LLMs

Innovation

• Using collaborative autoethnography to evaluate LLMs
• Assessing methodological rigor and practical usefulness
• Interpreting results through technical lens of LLMs