Repurposing the scientific literature with vision-language models

2025-02-26

AI Summary

General-purpose vision-language models (VLMs) lack domain-specific expertise in neurosurgery, limiting their clinical and academic utility. Method: We introduce NeuroPubs—the first neurosurgery-specific multimodal database—curated from peer-reviewed journals and used to train a domain-customized VLM. Our approach integrates multimodal alignment, curriculum learning, and domain-adaptive fine-tuning to transform scholarly literature into an AI-accessible, multimodal scientific knowledge base. Contribution/Results: The model enables three high-value applications: (1) publication-grade figure caption generation (70% output rated production-ready), (2) ABNS-style medical question synthesis (54% passed expert authenticity evaluation; 89,587 questions generated), and (3) clinical diagnostic support (CNS-Obsidian demonstrated non-inferiority to GPT-4o, *p* = 0.1154). All outputs underwent rigorous validation via expert review and clinical blind testing, establishing a reproducible paradigm for discipline-specific VLM development.

Technology Category

• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Neuro-Symbolic Learning

Application Category

• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Research in AI for Science often focuses on using AI technologies to augment components of the scientific process, or in some cases, the entire scientific method; how about AI for scientific publications? Peer-reviewed journals are foundational repositories of specialized knowledge, written in discipline-specific language that differs from general Internet content used to train most large language models (LLMs) and vision-language models (VLMs). We hypothesized that by combining a family of scientific journals with generative AI models, we could invent novel tools for scientific communication, education, and clinical care. We converted 23,000 articles from Neurosurgery Publications into a multimodal database - NeuroPubs - of 134 million words and 78,000 image-caption pairs to develop six datasets for building AI models. We showed that the content of NeuroPubs uniquely represents neurosurgery-specific clinical contexts compared with broader datasets and PubMed. For publishing, we employed generalist VLMs to automatically generate graphical abstracts from articles. Editorial board members rated 70% of these as ready for publication without further edits. For education, we generated 89,587 test questions in the style of the ABNS written board exam, which trainee and faculty neurosurgeons found indistinguishable from genuine examples 54% of the time. We used these questions alongside a curriculum learning process to track knowledge acquisition while training our 34 billion-parameter VLM (CNS-Obsidian). In a blinded, randomized controlled trial, we demonstrated the non-inferiority of CNS-Obsidian to GPT-4o (p = 0.1154) as a diagnostic copilot for a neurosurgical service. Our findings lay a novel foundation for AI with Science and establish a framework to elevate scientific communication using state-of-the-art generative artificial intelligence while maintaining rigorous quality standards.

Problem

• AI for scientific publications enhancement
• Multimodal database for neurosurgery content
• Generative AI for scientific communication tools

Innovation

• Vision-language models repurposed
• Multimodal database for AI training
• Generated graphical abstracts automatically