Style or Content? Evaluating Style Classifiers with Controlled Content Overlap

2026-06-05

AI Summary

This work addresses the prevalent reliance of existing style classifiers on content cues correlated with style labels and the lack of effective methods to quantify such dependence. To tackle this, the authors propose an information-theoretic framework for controllable content overlap, leveraging parallel Bible translations to construct a tunable overlap parameter α that systematically modulates the degree of content sharing across style categories. Through experiments involving RoBERTa-based classifiers, mutual information estimation, cross-style content retrieval probes, and training dynamics analysis, they demonstrate that model performance degrades significantly under low α, while high-α models exhibit greater robustness. Moreover, content recoverability decreases as α increases, confirming the framework’s effectiveness in disentangling style learning from content-based shortcuts.

Abstract

Style classifiers can use content cues that correlate with style labels in naturally collected data, yet we lack a systematic way to measure this reliance. We study this problem with a controlled content overlap setup built on parallel Bible translations. Specifically, we define the overlap parameter $α$ as the normalized residual of mutual information between content identity and style label, so that it measures how much content is shared across style classes: from no shared content ($α=0$) to fully shared content ($α=1$). Cross-overlap evaluation of RoBERTa-based classifiers shows that low-overlap models degrade when content cues are removed, while high-overlap models transfer more robustly. A cross-style content retrieval probe further shows that content becomes less recoverable as $α$ increases, with training dynamics showing this removal occurs gradually. Together, these results suggest that controlled overlap provides a simple diagnostic for separating style learning from content shortcuts.

Problem

• style classification
• content cues
• controlled content overlap
• style-content disentanglement
• mutual information

Innovation

• controlled content overlap
• style classification
• content-style disentanglement
• mutual information
• RoBERTa-based classifier