Many Heads Are Better Than One: Improved Scientific Idea Generation by A LLM-Based Multi-Agent System

2024-10-12

AI Summary

Current large language models (LLMs) show initial promise in scientific hypothesis generation and experimental design but fail to emulate the collaborative problem-solving dynamics of expert research teams. To address this, we propose VirSci—the first multi-agent system grounded in role-based division of labor and dynamic feedback. Built atop open-source LLMs, VirSci integrates prompt engineering, role-specialized agent design, iterative critical evaluation, and consensus-driven coordination to establish a scalable framework for scientific collaboration. Its core contribution lies in formalizing authentic research team workflows into a staged knowledge co-creation paradigm, thereby overcoming the limitations of isolated, single-model generation. Experiments demonstrate that VirSci surpasses state-of-the-art methods across multiple scientific creativity benchmarks, achieving a 23.6% improvement in novelty; moreover, collaboration depth exhibits a statistically significant positive correlation with idea quality.

Technology Category

• Multiagent Systems: Teamwork
• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Agentic search
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

The rapid advancement of scientific progress requires innovative tools that can accelerate knowledge discovery. Although recent AI methods, particularly large language models (LLMs), have shown promise in tasks such as hypothesis generation and experimental design, they fall short of replicating the collaborative nature of real-world scientific practices, where diverse experts work together in teams to tackle complex problems. To address the limitations, we propose an LLM-based multi-agent system, i.e., Virtual Scientists (VirSci), designed to mimic the teamwork inherent in scientific research. VirSci organizes a team of agents to collaboratively generate, evaluate, and refine research ideas. Through comprehensive experiments, we demonstrate that this multi-agent approach outperforms the state-of-the-art method in producing novel scientific ideas. We further investigate the collaboration mechanisms that contribute to its tendency to produce ideas with higher novelty, offering valuable insights to guide future research and illuminating pathways toward building a robust system for autonomous scientific discovery. The code is available at https://github.com/open-sciencelab/Virtual-Scientists.

Problem

• Enhance scientific idea generation
• Mimic real-world scientific collaboration
• Improve novelty in research ideas

Innovation

• LLM-based multi-agent system
• Teamwork simulation in research
• Enhanced scientific idea generation