SPADER: Step-wise Peer Advantage with Diversity-Aware Exploration Rewards for Multi-Answer Question Answering

2026-05-30

AI Summary

This work addresses the challenges of fine-grained credit assignment and exploration-reward alignment in multi-answer question answering under prolonged tool usage. To this end, the authors propose SPADER, a novel framework that introduces a critic-free step-wise peer advantage mechanism to enable step-level credit assignment across parallel reasoning trajectories. Additionally, SPADER incorporates a dynamic, diversity-aware exploration reward based on answer rarity, effectively incentivizing the discovery of long-tail answers. Evaluated on multiple benchmarks—including QAMPARI, Mintaka, WebQSP, and QUEST—SPADER achieves substantial improvements in recall and F1 scores, outperforming existing approaches such as prompt engineering, outcome-supervised reinforcement learning, and current step-supervised methods.

Abstract

Large language models are increasingly deployed as tool-augmented agents to acquire information beyond parametric knowledge. While recent work has improved long-horizon tool-use reasoning, most approaches focus on tasks with a single correct answer. In contrast, many real-world queries require discovering a comprehensive set of valid answers, a setting known as Multi-Answer QA. This setting raises two challenges: fine-grained credit assignment over long search trajectories and reward alignment for sustained exploration beyond easy high-frequency entities. We propose SPADER, a reinforcement learning framework for long-horizon tool use in Multi-Answer QA. SPADER includes Step-wise Peer Advantage (SPA), a critic-free step-level credit assignment mechanism that aligns parallel trajectories by decision step and estimates advantages from peer returns. It also includes a diversity-aware exploration reward that promotes long-tail entity discovery by upweighting rare findings and downweighting redundant ones. Experiments on QAMPARI, Mintaka, WebQSP, and QUEST show that SPADER generally improves recall and overall F1 over prompting-based agents, outcome-supervised RL methods, and recent step-level supervision approaches. Our code and model weights are available at https://github.com/KhanCold/spader.

Problem

• Multi-Answer QA
• credit assignment
• exploration reward
• long-horizon reasoning
• answer diversity

Innovation

• Step-wise Peer Advantage
• Diversity-Aware Exploration
• Multi-Answer QA
• Reinforcement Learning
• Long-Horizon Tool Use