🤖 AI Summary
Existing PDF parsing tools lack systematic empirical evaluation on non-academic documents (e.g., patents, financial reports, legal contracts). This paper presents the first comprehensive benchmarking study across six real-world document categories using the DocLayNet dataset, evaluating ten state-of-the-art tools—including PyMuPDF, Nougat, and TATR—on text extraction and table detection. Results reveal strong document-type dependency in tool performance: learning-based models (Nougat, TATR) consistently outperform rule-based approaches on complex layouts; PyMuPDF achieves the best overall text extraction accuracy; TATR leads in table detection across four document types; Camelot exhibits superior adaptability to tender documents; and Nougat excels on scientific and patent documents. By establishing a rigorous, domain-diverse evaluation framework, this work fills a critical gap in empirical assessment of PDF parsing for non-academic domains and provides evidence-based guidance for context-aware tool selection.
📝 Abstract
PDF is one of the most prominent data formats, making PDF parsing crucial for information extraction and retrieval, particularly with the rise of RAG systems. While various PDF parsing tools exist, their effectiveness across different document types remains understudied, especially beyond academic papers. Our research aims to address this gap by comparing 10 popular PDF parsing tools across 6 document categories using the DocLayNet dataset. These tools include PyPDF, pdfminer-six, PyMuPDF, pdfplumber, pypdfium2, Unstructured, Tabula, Camelot, as well as the deep learning-based tools Nougat and Table Transformer(TATR). We evaluated both text extraction and table detection capabilities. For text extraction, PyMuPDF and pypdfium generally outperformed others, but all parsers struggled with Scientific and Patent documents. For these challenging categories, learning-based tools like Nougat demonstrated superior performance. In table detection, TATR excelled in the Financial, Patent, Law&Regulations, and Scientific categories. Table detection tool Camelot performed best for tender documents, while PyMuPDF performed superior in the Manual category. Our findings highlight the importance of selecting appropriate parsing tools based on document type and specific tasks, providing valuable insights for researchers and practitioners working with diverse document sources.