🤖 AI Summary
Existing performance-degradation-based methods for modality contribution assessment struggle to disentangle a modality’s unique information content from its synergistic interaction effects—particularly under cross-attention architectures, where they face fundamental limitations. To address this, we propose the first representation-level quantification framework grounded in Partial Information Decomposition (PID), which rigorously decomposes multimodal representations into unique, redundant, and synergistic information components. Our method integrates PID theory with the Iterative Proportional Fitting Procedure (IPFP), enabling layer-wise and cross-dataset contribution inference without retraining. Experiments demonstrate that our framework substantially enhances both the interpretability and accuracy of contribution analysis. It provides a principled, fine-grained separation of each modality’s independent and interactive contributions, establishing a new paradigm for multimodal model diagnosis and architecture design.
📝 Abstract
Quantifying modality contributions in multimodal models remains a challenge, as existing approaches conflate the notion of contribution itself. Prior work relies on accuracy-based approaches, interpreting performance drops after removing a modality as indicative of its influence. However, such outcome-driven metrics fail to distinguish whether a modality is inherently informative or whether its value arises only through interaction with other modalities. This distinction is particularly important in cross-attention architectures, where modalities influence each other's representations. In this work, we propose a framework based on Partial Information Decomposition (PID) that quantifies modality contributions by decomposing predictive information in internal embeddings into unique, redundant, and synergistic components. To enable scalable, inference-only analysis, we develop an algorithm based on the Iterative Proportional Fitting Procedure (IPFP) that computes layer and dataset-level contributions without retraining. This provides a principled, representation-level view of multimodal behavior, offering clearer and more interpretable insights than outcome-based metrics.