Predicting Multi-View Rashomon Representation: Can We Learn Where Models Disagree?

2026-09-30

AI Summary

This study addresses the challenge of reliability assessment in foundation models arising from inconsistent representations. We propose a novel method to predict multi-view Rashomon representation divergence from a single-model perspective. By leveraging nearest-neighbor comparisons and a lightweight regression predictor, our approach efficiently estimates input-dependent cross-model representational discrepancies without requiring access to multiple models. This work is the first to reveal the predictability and generalizability of cross-model divergence within a single representation space. Consequently, it enables low-cost reliability assessment for foundation models and establishes a new paradigm for understanding the consistency and robustness of multi-view representation spaces.

Abstract

Foundation models are increasingly adopted across a wide range of applications, often serving as core blocks within AI systems. Yet different foundation models may encode the same input from multiple different views, leading to substantial representation disagreement, which we term Rashomon Representation. Such disagreement often signals inputs that a given model encodes in a way inconsistent with other models, offering a valuable yet underexplored signal for input reliability estimation. While prior work has largely focused on measuring disagreement across multiple models with a representation set, we instead focus on predicting disagreement from a single representation. We hypothesize that this disagreement follows some consistent, input-dependent patterns rather than occurring at random. To test this, we quantify disagreement by comparing each sample's nearest neighbors across different models' representation spaces, then train a lightweight predictor that estimates disagreement from a single model's representation. At inference time, given a new input, the predictor uses that input's representation to tell whether it aligns with or diverges from those of other models. Extensive experiments across diverse foundation models and datasets show that representational disagreement is indeed input-dependent, predictable, and generalizable, enabling efficient reliability estimation of foundation models.

Problem

• Foundation Models
• Representation Disagreement
• Rashomon Representation
• Reliability Estimation
• Multi-View

Innovation

• Rashomon Representation
• Foundation Models
• Representation Disagreement
• Reliability Estimation
• Lightweight Predictor