Shortcut Learning in Legal Judgment Prediction: Empirical Evidence from the UK Employment Tribunal

2026-07-05

AI Summary

This study addresses the pervasive issue of shortcut learning in legal judgment prediction models, which often exploit spurious surface cues from post-hoc judicial texts, leading to inflated performance estimates. Leveraging 33,158 claim forms from UK Employment Tribunals and LLM-generated summaries, the work presents the first empirical evidence of such data leakage in this task. By integrating TF-IDF-based interpretable classifiers with black-box large language models and human-annotated leakage indicators, the authors systematically identify, mask, and retrain on contaminated features. Experiments reveal that as few as 4% of leakage-prone features can surpass human expert prediction accuracy; however, their removal results in only a marginal drop in Macro-F1, demonstrating the model’s robust capacity for substantive reasoning. The study advocates a new paradigm: treating post-decision texts as potential contamination sources requiring proactive auditing and debiasing.

Abstract

Current Legal Judgment Prediction (LJP) is constrained by its reliance on post-hoc judicial materials, increasing the likelihood that models perform retrospective classification rather than true forecasting. This paper empirically investigates shortcut learning in this context by studying claim-level outcome prediction in UK Employment Tribunal (UKET) decisions. Using a corpus of 33,158 individual claims, we predict outcomes from claim texts and LLM-extracted case summaries, evaluating models ranging from interpretable TF-IDF-based classifiers to black-box LLMs. While headline predictive performance figures appear strong, we demonstrate that such performance in LJP systems trained on post-hoc judicial text can be driven by the retrospective nature of the source material. Stratifying the test data by human judgments of leakage reveals that performance increases where outcome-revealing cues are embedded in the narrative. Moreover, a model trained on just the 4% of features identified as leakage achieves high performance, outperforming human experts. These findings substantiate concerns that LJP performance may be exaggerated by linguistic artefacts. Yet this vulnerability is not fatal to the research agenda. Instead, post-hoc judgments might be treated as potentially contaminated texts, requiring active auditing. Retraining models after masking leakage features results in only a negligible reduction in Macro-F1. Hence, while models will opportunistically exploit shortcuts when available, they remain capable of extracting useful predictive signals when these artefacts are removed.

Problem

• Legal Judgment Prediction
• Shortcut Learning
• Outcome Leakage
• Retrospective Classification
• Employment Tribunal

Innovation

• shortcut learning
• legal judgment prediction
• information leakage
• post-hoc judicial text
• model auditing