🤖 AI Summary
Medical literature is vast and structurally complex, impeding efficient knowledge acquisition by researchers. To address this, we propose an end-to-end medical text named entity recognition (NER) system featuring a dual-engine architecture that synergistically integrates deep learning with dynamic dictionary matching. Specifically, it employs fine-tuned SciBERT/BERT-based NER models augmented with rule-enhanced, extensible dictionary lookup to identify five entity types—including diseases, genes, and chemicals. The system further incorporates entity disambiguation, co-occurrence relation graph construction, and publication-grade ranked output. It supports large-scale corpora such as PubMed and CORD-19, and provides a Python API alongside interactive visualization for validation. Evaluated on a dataset of 764,000 abstracts covering autophagy and COVID-19, the system achieves high precision and sub-minute inference latency. The source code is publicly available and has been deployed in real-world biomedical research workflows.
📝 Abstract
Medical research generates a large number of publications with the PubMed database already containing >35 million research articles. Integration of the knowledge scattered across this large body of literature could provide key insights into physiological mechanisms and disease processes leading to novel medical interventions. However, it is a great challenge for researchers to utilize this information in full since the scale and complexity of the data greatly surpasses human processing abilities. This becomes especially problematic in cases of extreme urgency like the COVID-19 pandemic. Automated text mining can help extract and connect information from the large body of medical research articles. The first step in text mining is typically the identification of specific classes of keywords (e.g., all protein or disease names), so called Named Entity Recognition (NER). Here we present an end-to-end pipeline for NER of typical entities found in medical research articles, including diseases, cells, chemicals, genes/proteins, and species. The pipeline can access and process large medical research article collections (PubMed, CORD-19) or raw text and incorporates a series of deep learning models fine-tuned on the HUNER corpora collection. In addition, the pipeline can perform dictionary-based NER related to COVID-19 and other medical topics. Users can also load their own NER models and dictionaries to include additional entities. The output consists of publication-ready ranked lists and graphs of detected entities and files containing the annotated texts. An associated script allows rapid inspection of the results for specific entities of interest. As model use cases, the pipeline was deployed on two collections of autophagy-related abstracts from PubMed and on the CORD19 dataset, a collection of 764 398 research article abstracts related to COVID-19.