RbtAct: Rebuttal as Supervision for Actionable Review Feedback Generation

2026-03-10

AI Summary

This work addresses the limited actionability of current AI-generated peer reviews, which often fail to provide authors with concrete guidance for revision. To tackle this issue, the authors propose RbtAct, a novel approach that formulates paragraph-level review generation under reviewer perspective conditioning as a new task and leverages authors’ rebuttals to reviews as implicit supervision signals. Built upon the Llama-3.1-8B-Instruct model, RbtAct integrates supervised fine-tuning with preference optimization guided by rebuttal pairs, and introduces RMR-75K, a dataset comprising 75K review-rebuttal samples. Experimental results demonstrate that RbtAct significantly enhances the actionability and specificity of generated reviews, as evaluated by both human experts and LLM-as-a-judge metrics, while preserving content relevance and factual consistency.

Technology Category

• Natural Language Processing: Generation
• Humans and AI: Human-AI Collaboration / Human-AI Teaming
• Knowledge Representation and Reasoning: Action, Change, and Causality

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large language models (LLMs) are increasingly used across the scientific workflow, including to draft peer-review reports. However, many AI-generated reviews are superficial and insufficiently actionable, leaving authors without concrete, implementable guidance and motivating the gap this work addresses. We propose RbtAct, which targets actionable review feedback generation and places existing peer review rebuttal at the center of learning. Rebuttals show which reviewer comments led to concrete revisions or specific plans, and which were only defended. Building on this insight, we leverage rebuttal as implicit supervision to directly optimize a feedback generator for actionability. To support this objective, we propose a new task called perspective-conditioned segment-level review feedback generation, in which the model is required to produce a single focused comment based on the complete paper and a specified perspective such as experiments and writing. We also build a large dataset named RMR-75K that maps review segments to the rebuttal segments that address them, with perspective labels and impact categories that order author uptake. We then train the Llama-3.1-8B-Instruct model with supervised fine-tuning on review segments followed by preference optimization using rebuttal derived pairs. Experiments with human experts and LLM-as-a-judge show consistent gains in actionability and specificity over strong baselines while maintaining grounding and relevance.

Problem

• actionable feedback
• peer review
• scientific writing
• review generation
• rebuttal

Innovation

• rebuttal-as-supervision
• actionable feedback generation
• perspective-conditioned review
• preference optimization
• RMR-75K dataset