Beyond Routing Weights: Faithful Response-Level Interpretation of Mixture-of-Experts Reward Models via Contribution Contrast

2026-07-31

AI Summary

Existing sparse Mixture-of-Experts (MoE) reward models rely solely on routing weights to interpret expert behavior, which fails to reveal the underlying rationale for their judgments of responses, resulting in incomplete explanations. This work proposes Contribution Contrast (CoCo), a novel method that analyzes the difference in expert contributions between chosen and rejected response pairs to jointly model routing and preference signals. CoCo is the first approach to deliver response-level explanations that are faithful, coherent, and expert-specific. Experimental results demonstrate that CoCo significantly outperforms existing explanation methods—including routing weights, score-based baselines, and sparse autoencoders—in both automatic and human evaluations, while maintaining competitive reward modeling accuracy.

Abstract

Reward models are central to learning from human preferences, yet identifying what drives their predictions remains challenging. Recent sparse Mixture-of-Experts (MoE) reward models seek to improve interpretability by routing prompts to specialized experts and characterizing experts through examples with high routing weights. However, routing weights only reveal which prompts an expert $\textit{receives}$, not how it $\textit{judges}$ responses, providing only a partial account of expert behavior. We therefore propose $\textbf{Co}$ntribution-$\textbf{Co}$ntrast ($\textbf{CoCo}$) response-level interpretation, which faithfully characterizes experts' roles using chosen-rejected response pairs with the largest contribution contrasts, jointly capturing routing and preference behavior. Across automatic and human evaluations, CoCo yields more coherent, faithful, and specialized interpretations than router-based, score-based, and sparse autoencoder-based alternatives while maintaining competitive reward modeling accuracy. To the best of our knowledge, this is the first systematic study of interpretation methods for MoE reward models.

Problem

• Mixture-of-Experts
• reward models
• interpretability
• routing weights
• response-level interpretation

Innovation

• Mixture-of-Experts
• reward models
• interpretability
• Contribution-Contrast
• response-level interpretation