AirQA: A Comprehensive QA Dataset for AI Research with Instance-Level Evaluation

2025-09-21

AI Summary

Scientific literature question answering lacks high-quality evaluation benchmarks and interactive training data. Method: This paper introduces AirQA—the first multi-task, multimodal scientific paper QA dataset tailored for AI research, comprising 13,948 papers and 1,246 questions, enabling instance-level fine-grained evaluation. We propose ExTrActor, an automated instruction synthesis framework that generates high-quality, multi-turn interactive trajectories without human intervention, integrating multi-agent collaboration, tool invocation, and interactive retrieval. Contribution/Results: Experiments show that state-of-the-art open- and closed-source models achieve limited performance on AirQA, confirming its strong challenge level. ExTrActor significantly enhances small language models’ capability in multi-turn tool usage, approaching the performance of large models. The code, dataset, and interaction trajectories are fully open-sourced.

Technology Category

• Natural Language Processing: Question Answering
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

The growing volume of academic papers has made it increasingly difficult for researchers to efficiently extract key information. While large language models (LLMs) based agents are capable of automating question answering (QA) workflows for scientific papers, there still lacks a comprehensive and realistic benchmark to evaluate their capabilities. Moreover, training an interactive agent for this specific task is hindered by the shortage of high-quality interaction trajectories. In this work, we propose AirQA, a human-annotated comprehensive paper QA dataset in the field of artificial intelligence (AI), with 13,948 papers and 1,246 questions, that encompasses multi-task, multi-modal and instance-level evaluation. Furthermore, we propose ExTrActor, an automated framework for instruction data synthesis. With three LLM-based agents, ExTrActor can perform example generation and trajectory collection without human intervention. Evaluations of multiple open-source and proprietary models show that most models underperform on AirQA, demonstrating the quality of our dataset. Extensive experiments confirm that ExTrActor consistently improves the multi-turn tool-use capability of small models, enabling them to achieve performance comparable to larger ones.

Problem

• Lack of comprehensive benchmark for evaluating LLM-based QA on scientific papers
• Shortage of high-quality interaction trajectories for training interactive agents
• Difficulty extracting key information from growing volume of academic papers

Innovation

• Human-annotated dataset for AI paper QA
• Automated framework for instruction data synthesis
• LLM-based agents for trajectory collection