Accepted with Minor Revisions: Value of AI-Assisted Scientific Writing

2025-11-16

AI Summary

This study investigates the assistive efficacy of large language models (LLMs) in scientific abstract writing, focusing on factual accuracy, multimodal integration, and domain-specific adaptability. A 2×2 randomized controlled trial was conducted within a simulated conference review setting, incorporating behavioral economics–informed incentives and multivariate statistical analysis to compare editor behavior and acceptance outcomes for AI-generated versus human-written abstracts. Key contributions: (1) Editor decisions were driven primarily by source attribution—specifically, whether AI generation was disclosed—rather than objective quality metrics; (2) AI-generated abstracts required only minimal editing to achieve parity with human-authored counterparts; (3) transparent source labeling significantly reduced disparities in editing effort; and (4) fine-grained stylistic calibration increased acceptance probability by 12.3%. These findings provide empirical support and behavioral mechanism insights for the credible, high-stakes deployment of LLMs in scholarly communication.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Humans and AI: Learning Human Values and PreferencesCognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

Economics, Online Markets and Human Computation: LLM based quality controls for crowd workSemantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactionsUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models have seen expanding application across domains, yet their effectiveness as assistive tools for scientific writing -- an endeavor requiring precision, multimodal synthesis, and domain expertise -- remains insufficiently understood. We examine the potential of LLMs to support domain experts in scientific writing, with a focus on abstract composition. We design an incentivized randomized controlled trial with a hypothetical conference setup where participants with relevant expertise are split into an author and reviewer pool. Inspired by methods in behavioral science, our novel incentive structure encourages authors to edit the provided abstracts to an acceptable quality for a peer-reviewed submission. Our 2x2 between-subject design expands into two dimensions: the implicit source of the provided abstract and the disclosure of it. We find authors make most edits when editing human-written abstracts compared to AI-generated abstracts without source attribution, often guided by higher perceived readability in AI generation. Upon disclosure of source information, the volume of edits converges in both source treatments. Reviewer decisions remain unaffected by the source of the abstract, but bear a significant correlation with the number of edits made. Careful stylistic edits, especially in the case of AI-generated abstracts, in the presence of source information, improve the chance of acceptance. We find that AI-generated abstracts hold potential to reach comparable levels of acceptability to human-written ones with minimal revision, and that perceptions of AI authorship, rather than objective quality, drive much of the observed editing behavior. Our findings reverberate the significance of source disclosure in collaborative scientific writing.

Problem

• Evaluating LLMs as assistive tools for scientific abstract writing
• Measuring how AI source disclosure affects editing behavior and acceptance
• Assessing perception biases versus objective quality in AI-assisted writing

Innovation

• Randomized controlled trial with incentive structure
• 2x2 design tests source and disclosure effects
• Measures editing behavior and acceptance correlation