Automated Review Generation Method Based on Large Language Models

2024-07-30

arXiv.org

AI Summary

To address low literature review efficiency, high domain-knowledge barriers, and prominent hallucination risks in large language models (LLMs), this work proposes a fully automated LLM-based literature review generation method. We introduce a statistically validated, multi-layer quality control framework that reduces factual hallucination to below 0.5% (95% confidence interval), while ensuring citation completeness and enabling domain-agnostic prompt engineering. The system supports cross-disciplinary deployment without requiring domain expertise and is distributed as a one-click Windows desktop application. Empirical evaluation in propane dehydrogenation catalysis demonstrates its efficacy: it processes 343 papers in seconds, covering 35 thematic categories; extended analysis of 1,041 papers achieves expert-validated accuracy and citation integrity.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Literature research, vital for scientific work, faces the challenge of surging information volumes exceeding researchers' processing capabilities. We present an automated review generation method based on large language models (LLMs) to overcome efficiency bottlenecks and reduce cognitive load. Our statistically validated evaluation framework demonstrates that the generated reviews match or exceed manual quality, offering broad applicability across research fields without requiring users' domain knowledge. Applied to propane dehydrogenation (PDH) catalysts, our method swiftly analyzed 343 articles, averaging seconds per article per LLM account, producing comprehensive reviews spanning 35 topics, with extended analysis of 1041 articles providing insights into catalysts' properties. Through multi-layered quality control, we effectively mitigated LLMs' hallucinations, with expert verification confirming accuracy and citation integrity while demonstrating hallucination risks reduced to below 0.5% with 95% confidence. Released Windows application enables one-click review generation, enhancing research productivity and literature recommendation efficiency while setting the stage for broader scientific explorations.

Problem

• Scientific Literature Review
• Information Processing
• Efficiency Enhancement

Innovation

• Large Language Models
• Automated Literature Review
• High-efficiency Research