A Multi-Dimensional Evaluation of Explainability in Media Bias Detection

2026-07-22

AI Summary

This study addresses the critical need for interpretability in automated media bias detection, arguing that reliable deployment requires not only high predictive accuracy but also transparent reasoning aligned with human judgment. The authors propose the first multidimensional evaluation framework encompassing predictive performance, explanation plausibility, and mechanistic faithfulness. Using the BABE dataset, they systematically assess BERT and RoBERTa (base and large variants) and introduce attention-supervised fine-tuning to better align model explanations with expert rationales. Through counterfactual masking tests and circuit analysis, they demonstrate that model architecture substantially influences explanation quality and that model scale alone does not determine mechanistic compressibility. The findings underscore the necessity of evaluating interpretability across these three distinct dimensions to achieve a comprehensive understanding.

Abstract

Detecting media bias automatically is difficult because biased framing is often subtle, yet in domains such as news analysis, accurate predictions alone are insufficient without explanations that reflect the model's underlying reasoning. We present a multi-dimensional evaluation of explainability in encoder-based media bias detection using the Bias Annotations By Experts (BABE) dataset. Specifically, we study BERT and RoBERTa as classifiers (base and large variants) along three complementary axes: predictive performance, explanation plausibility (token-level alignment with expert rationales), and mechanistic faithfulness (whether compact sets of attention heads recover predictive signal under counterfactual rationale masking). To induce variation in plausibility, we additionally investigate attention-supervised finetuning, which incorporates expert rationale annotations as an auxiliary training signal. Attention supervision serves as an intervention on attribution plausibility, while the effectiveness of attribution methods varies substantially across architectures. Circuit analysis further reveals substantial variation in mechanistic recoverability across architectures, suggesting that model scale alone does not determine circuit compressibility. Taken together, our findings suggest that predictive performance, attribution plausibility, and mechanistic faithfulness characterize different aspects of model behavior and should be evaluated separately when studying explainability in media bias detection.

Problem

• media bias detection
• explainability
• attribution plausibility
• mechanistic faithfulness
• predictive performance

Innovation

• explainability evaluation
• attention supervision
• mechanistic faithfulness
• media bias detection
• counterfactual rationale masking