🤖 AI Summary
This study addresses the time- and resource-intensive bottleneck of data extraction in scoping reviews by proposing an LLM-driven automation framework leveraging Claude 3.5 Sonnet and a structured review protocol. Methodologically, the protocol is directly embedded into system prompts, and an LLM-in-the-loop feedback mechanism is introduced to iteratively refine protocol design. Results show high accuracy (83.3%–100%) for simple, structured fields (e.g., author, year), but sharp declines for complex, subjective constructs (e.g., study aims, limitations)—with accuracy dropping to 9.6%–15.8% and overall F1 < 40%, primarily due to low recall (<25%). Protocol embedding improves inter-extraction consistency, and LLM feedback supports effective protocol refinement; however, current models exhibit insufficient reliability for semantic abstraction tasks. The study underscores the necessity of multi-dimensional performance evaluation and human-in-the-loop validation, offering both a methodological framework and pragmatic caution for deploying LLMs in evidence synthesis.
📝 Abstract
The data extraction stages of reviews are resource-intensive, and researchers may seek to expediate data extraction using online (large language models) LLMs and review protocols. Claude 3.5 Sonnet was used to trial two approaches that used a review protocol to prompt data extraction from 10 evidence sources included in a case study scoping review. A protocol-based approach was also used to review extracted data. Limited performance evaluation was undertaken which found high accuracy for the two extraction approaches (83.3% and 100%) when extracting simple, well-defined citation details; accuracy was lower (9.6% and 15.8%) when extracting more complex, subjective data items. Considering all data items, both approaches had precision >90% but low recall (<25%) and F1 scores (<40%). The context of a complex scoping review, open response types and methodological approach likely impacted performance due to missed and misattributed data. LLM feedback considered the baseline extraction accurate and suggested minor amendments: four of 15 (26.7%) to citation details and 8 of 38 (21.1%) to key findings data items were considered to potentially add value. However, when repeating the process with a dataset featuring deliberate errors, only 2 of 39 (5%) errors were detected. Review-protocol-based methods used for expediency require more robust performance evaluation across a range of LLMs and review contexts with comparison to conventional prompt engineering approaches. We recommend researchers evaluate and report LLM performance if using them similarly to conduct data extraction or review extracted data. LLM feedback contributed to protocol adaptation and may assist future review protocol drafting.