CAuSE: Decoding Multimodal Classifiers using Faithful Natural Language Explanation

2025-12-07

AI Summary

Multimodal classifiers lack faithful and interpretable natural language explanations (NLEs), and existing methods fail to accurately reflect their internal decision-making logic. To address this, we propose CAuSE, the first framework that integrates causal abstraction modeling with interleaved intervention training to mechanistically emulate the model’s causal reasoning process and generate high-fidelity NLEs. We introduce a novel causal fidelity metric to quantitatively evaluate explanation quality. Experiments across multiple benchmark datasets demonstrate that CAuSE significantly outperforms state-of-the-art methods, achieving substantial gains in causal fidelity. Qualitative analysis and systematic error analysis further confirm the reasonableness and robustness of its explanations. The code is publicly available.

Technology Category

• Machine Learning: Causal Learning
• Reasoning under Uncertainty: Causality
• Computer Vision: Interpretability, Explainability, and Transparency

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Multimodal classifiers function as opaque black box models. While several techniques exist to interpret their predictions, very few of them are as intuitive and accessible as natural language explanations (NLEs). To build trust, such explanations must faithfully capture the classifier's internal decision making behavior, a property known as faithfulness. In this paper, we propose CAuSE (Causal Abstraction under Simulated Explanations), a novel framework to generate faithful NLEs for any pretrained multimodal classifier. We demonstrate that CAuSE generalizes across datasets and models through extensive empirical evaluations. Theoretically, we show that CAuSE, trained via interchange intervention, forms a causal abstraction of the underlying classifier. We further validate this through a redesigned metric for measuring causal faithfulness in multimodal settings. CAuSE surpasses other methods on this metric, with qualitative analysis reinforcing its advantages. We perform detailed error analysis to pinpoint the failure cases of CAuSE. For replicability, we make the codes available at https://github.com/newcodevelop/CAuSE

Problem

• Generates faithful natural language explanations for multimodal classifiers
• Measures causal faithfulness in multimodal settings
• Generalizes across datasets and models via causal abstraction

Innovation

• Generates faithful natural language explanations for classifiers
• Uses causal abstraction via interchange intervention training
• Introduces new metric for measuring causal faithfulness