Unraveling the Ai2 Asta Scholarly Research Assistant Citation System

2026-06-06

AI Summary

This study addresses the critical lack of empirical investigation into the transparency, stability, and potential biases of citation mechanisms in AI-powered academic assistants. It presents the first systematic evaluation of Ai2 Asta’s citation behavior during literature review tasks, conducting two independent rounds of experiments with ten domain-specific queries to analyze in-text citation strength, reference composition, and consistency between retrieved documents and final citations. The findings reveal that, despite high citation intensity, the system exhibits substantial instability across repeated runs and a notable misalignment between retrieved sources and cited references, indicating an opaque underlying selection mechanism. This work provides essential empirical evidence concerning the reproducibility of AI-assisted research and the reliability of scientometric analyses derived from such tools.

Abstract

Despite the growing integration of Deep Research tools into academic workflows, empirical evidence on the operation, stability, and potential biases of their citation systems remains scarce. This study addresses this gap by evaluating the intensity, consistency, and bibliographic characteristics of references cited in the literature reports generated by Ai2 Asta, with the aim of understanding how its citation system operates and assessing its implications for scholarly communication. To this end, ten domain-specific queries were submitted to Asta's Summarise Literature feature, and two independent rounds of data collection were conducted. From each report, in-text citations, cited references, as well as other metrics related to the response process were extracted and examined. The results reveal high citation intensity, with reports integrating numerous in-text citations grounded in retrieved evidence and a diverse yet concentrated set of venues. However, notable instability is observed in the composition of cited references across identical queries, alongside a lack of concordance between retrieved documents and those ultimately cited, suggesting additional opaque selection mechanisms during report generation. These findings indicate that, while Ai2 Asta produces well-structured and quality reports, its instability and opacity in the citation process pose challenges in quantitative science studies due to their lack of reproducibility and transparency. Despite the restricted number of queries and disciplinary scope, the results offer valuable insights for researchers, bibliometricians, developers, and research evaluators seeking to understand, use or regulate AI-based scholarly assistants responsibly.

Problem

• citation system
• AI-based scholarly assistant
• reproducibility
• transparency
• bibliometric bias

Innovation

• citation system
• AI-based scholarly assistant
• reproducibility
• bibliometric bias
• research transparency