For What Reason? Interpreting Models' Encoding of Causation and Antithesis

2026-07-20

AI Summary

This study investigates how instruction-tuned large language models (LLaMA and Mistral) internally encode two types of contrastive discourse relations—causal and adversative—by framing discourse relation recognition as a next-token prediction task. Employing multiple interpretability methods, the authors analyze the dynamic representational processes across model layers. Their findings reveal that early layers initiate relation judgments around the middle of the input sequence, while middle layers finalize these decisions near the sequence end; the remaining layers primarily serve a relay function. Moreover, distinct layers exhibit representational preferences for specific relation types, highlighting the staged and asymmetric nature of discourse reasoning in large language models. This work provides fine-grained empirical evidence for understanding the internal mechanisms underlying linguistic inference in such models.

Abstract

Discourse relations provide document structure, critical to language understanding and enabling language model performance and ethicality. In this work, we investigate how instruction-tuned Transformer models (LLaMA and Mistral) encode discourse relations in English, with a particular focus on the contrasting relations of causation and antithesis. Framing the task as a next-token prediction task and applying a suite of interpretability techniques to test model internals, our findings show that certain early layers make predictive decisions at mid-sequence tokens, while some mid-level layers finalize their decisions closer to the last token. Most of the remaining layers primarily propagate earlier decisions rather than actively influencing them. Additionally, we observe that some layers exhibit a preference for one answer over alternatives, suggesting asymmetric representation of discourse-based reasoning.\footnote{Our code is available at https://github.com/abhidipbhattacharyya/causation_vs_antithesis}

Problem

• discourse relations
• causation
• antithesis
• language models
• model interpretability

Innovation

• discourse relations
• causation
• antithesis
• interpretability
• next-token prediction