AutoSupervision: Closing the Feedback Loop in Scientific Workflows with Grounded Revision Verification

2026-07-30

AI Summary

Current mechanisms struggle to verify whether revisions to scientific manuscripts substantively address peer reviewers’ concerns with supporting textual evidence. This work proposes AutoSupervision, a novel framework that leverages transparent peer review records from 56,000 papers in Nature Communications to construct a closed-loop evaluation system powered by large language models (e.g., GPT-5.5). The system automatically identifies reviewer concerns, assesses the effectiveness of author revisions, and locates supporting evidence within the revised text. Experimental results show that large language models achieve strong performance in identifying reviewer concerns (F1 = 0.754), yet evidence-based verification remains challenging, with the best current model attaining only an F1 score of 0.501. This study establishes a verifiable, evidence-driven paradigm for automated assessment of scientific manuscript revisions.

Abstract

Recent advances in large language models (LLMs) have enabled AI systems to assist scientific research and peer review. However, an essential capability for reliable AI-assisted scientific workflows remains underexplored: verifying whether reviewer feedback leads to meaningful and evidence-supported manuscript improvements. We introduce AutoSupervision, which evaluates whether scientific manuscript revisions genuinely address reviewer concerns through grounded evidence. AutoSupervision leverages transparent peer-review records as a natural source of supervision, where reviewer comments specify scientific concerns, author responses describe claimed resolutions, and revised manuscripts provide evidence of changes. Given reviewer comments, author responses, and revised manuscripts, models must characterize reviewer concerns, determine whether concerns have been addressed, and identify supporting manuscript evidence. We construct AutoSupervision from 56,000 Nature Communications articles and corresponding review records. Then we conducted experiments on LLMs, the ablation study, and the case study. Our results show that while LLMs perform well in characterizing reviewer concerns, with GPT-5.5 achieving a score of 0.754, evidence-based verification remains the primary bottleneck, with the best-performing model reaching only 0.501.

Problem

• scientific workflows
• peer review
• revision verification
• grounded evidence
• reviewer feedback

Innovation

• AutoSupervision
• grounded verification
• scientific peer review
• evidence-based revision
• LLM evaluation