BulkPR-Bench: Benchmarking Queue-Level Governance of Interacting Pull Requests

2026-08-03

AI Summary

This work addresses the limitations of existing code agent benchmarks, which fail to model interactions and dependencies among pull requests (PRs) in real-world scenarios and cannot guarantee safe merge orders. To tackle this, the authors propose BulkPR-Bench, the first executable benchmark for queue-level collaborative PR governance. It models PR dependencies and conflicts via a relational graph and introduces novel metrics—Relational Delivery Score (RDS) and Global Safe Yield (Global-SGY). Built on 581 PRs from 18 real repositories, the benchmark leverages state-level execution and hidden safety checks to validate the relational graph and compute the maximal safe subset. Experiments under a K=32 buffering protocol show that the best-performing model achieves an RDS of 66.6%, significantly outperforming sequential baselines (53.1%); however, full queue correctness is attained only eight times, with a peak critical relation recall of 57.7%, highlighting the substantial challenges remaining in batched PR coordination.

Abstract

Coding-agent benchmarks increasingly cover long-horizon, end-to-end, and interactive development, but typically retain one requested outcome or a fixed change sequence. Sequential policies can process a pull-request (PR) queue one candidate at a time, but when queued PRs interact, maximizing safe delivery can require jointly deciding which changes to merge and in what order. We introduce BulkPR-Bench, an executable benchmark in which an agent must recover consequential PR relations and return a large safe subset in executable order under a rolling-release protocol. The suite contains 581 newly authored candidate PRs on frozen snapshots of 18 real repositories. Registered state-by-state repository execution, including hidden safety checks, validates the gold relation graph; an exact oracle then computes the largest safe subset. Our primary metric, Relational Delivery Score (RDS), scores safe delivery and correct rejection over relation groups from the realized merge trace; Global Safety-Gated Yield (Global-SGY) separately measures strict delivery of the realized whole-queue plan. Under the buffered primary protocol with batch size $K=32$, the three highest RDS estimates among the six models are 66.6%, 62.0%, and 57.9%, compared with 53.1% for the strongest sequential baseline. Only 8 of 324 model runs complete a queue exactly. Critical-relation recall ranges from 35.2% to 57.7%, and diagnostic runs supplied with the gold relations show substantial remaining headroom. Gains on relation groups therefore do not yet translate into dependable whole-queue governance.

Problem

• pull request governance
• queue-level scheduling
• PR interactions
• safe merging
• relational dependencies

Innovation

• BulkPR-Bench
• pull request governance
• relational dependency
• executable benchmark
• rolling-release protocol