When Policies Change Probabilities: Modular Decision-Making for LLM Code Review

2026-08-02

AI Summary

This work addresses a critical yet previously overlooked issue in existing large language model (LLM)-based code review systems: the coupling of risk probability estimation with cost-sensitive decision strategies, which distorts probabilistic outputs by embedding policy biases. We empirically demonstrate this problem for the first time and introduce a modular decoupling framework that separates risk probability estimation, external evidence integration, and cost-sensitive decision-making into distinct components. By incorporating probability calibration, independent monitor-based score fusion, and a modular inference pipeline, our approach reduces average loss per issue by 0.073 at equivalent computational cost while accepting 58–68% of patches; in high-cost scenarios, it rejects all patches, substantially outperforming baseline methods. Moreover, the framework significantly enhances both the calibration accuracy of risk probabilities and the overall auditability of the system.

Abstract

LLM code reviewers often estimate patch risk and make approval decisions in one prompt. A probability should depend on evidence; costs should determine the action taken from it. We test whether four deployed reviewer interfaces preserve this separation using 15,792 responses on 720 candidate patches, with one that passed and one that failed an archived test harness for each of 360 repository issues. In matched calls with the patch and monitor evidence fixed, replacing an equal-cost policy with a 10:1 false-accept policy changes reported failure probabilities by 13.6 to 16.9 percentage points on average. For every reviewer, the actions returned under the high-cost prompt are worse than rejecting all patches. Applying the same high-cost rule to probabilities elicited under equal costs reduces loss for all four systems, showing that probability elicitation itself contributes to the excess loss. We also evaluate a modular pipeline that elicits risk without policy information, combines an independent monitor score, and applies costs in code. Relative to calibrated reviewer-only scores, the pipeline improves average probability accuracy and, at equal costs, reduces mean loss by .073 per issue while accepting 58 to 68% of patches. At 10:1, it accepts none and matches reject-all. Downstream policy can therefore change the probability it is meant to use, motivating separate evaluation of risk, outside evidence, and action.

Problem

• LLM code review
• probability elicitation
• decision policy
• modular decision-making
• risk estimation

Innovation

• modular decision-making
• probability elicitation
• cost-sensitive policy
• LLM code review
• risk calibration