Measuring IIA Violations in Similarity Choices with Bayesian Models

2025-08-20

AI Summary

This paper addresses the underexplored question of whether the Independence from Irrelevant Alternatives (IIA) assumption holds in human target-dependent similarity judgments—common in information retrieval and embedding learning. We propose the first systematic Bayesian framework for detecting and quantifying IIA violations. Innovatively, we introduce posterior predictive checking (PPC) into IIA testing, designing novel PPC statistics to assess population-level homogeneity and identifying context effects as the primary driver of IIA violations. Using both synthetically constructed and randomly generated choice-set data, and combining classical goodness-of-fit tests with PPC analysis, we empirically demonstrate significant and comparably sized IIA violations across datasets, while confirming strong behavioral homogeneity across participants. Our work establishes a more robust statistical foundation for modeling human similarity judgments, advancing both theoretical understanding and practical design of similarity-based systems.

Technology Category

• Knowledge Representation and Reasoning: Preferences
• Reasoning under Uncertainty: Relational Probabilistic Models
• Machine Learning: Learning Preferences or Rankings

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Similarity choice data occur when humans make choices among alternatives based on their similarity to a target, e.g., in the context of information retrieval and in embedding learning settings. Classical metric-based models of similarity choice assume independence of irrelevant alternatives (IIA), a property that allows for a simpler formulation. While IIA violations have been detected in many discrete choice settings, the similarity choice setting has received scant attention. This is because the target-dependent nature of the choice complicates IIA testing. We propose two statistical methods to test for IIA: a classical goodness-of-fit test and a Bayesian counterpart based on the framework of Posterior Predictive Checks (PPC). This Bayesian approach, our main technical contribution, quantifies the degree of IIA violation beyond its mere significance. We curate two datasets: one with choice sets designed to elicit IIA violations, and another with randomly generated choice sets from the same item universe. Our tests confirmed significant IIA violations on both datasets, and notably, we find a comparable degree of violation between them. Further, we devise a new PPC test for population homogeneity. Results show that the population is indeed homogenous, suggesting that the IIA violations are driven by context effects -- specifically, interactions within the choice sets. These results highlight the need for new similarity choice models that account for such context effects.

Problem

• Detecting IIA violations in similarity choice data
• Testing independence of irrelevant alternatives in human choices
• Quantifying context effects in similarity-based decision making

Innovation

• Bayesian Posterior Predictive Checks quantify IIA violation degree
• Two statistical methods test IIA in similarity choice settings
• New PPC test devised for population homogeneity assessment