VAR: Visual Analysis for Rashomon Set of Machine Learning Models' Performance

2025-07-30

AI Summary

Existing visualization methods lack effective horizontal comparative analysis for the Rashomon set—i.e., ensembles of structurally diverse models exhibiting comparable predictive performance. Method: This paper proposes VAR (Visualizing Aggregated Rashomon), a novel visualization framework featuring a horizontal multi-model juxtaposition layout—departing from conventional vertical, single-model paradigms. VAR integrates heatmaps (encoding feature importance and response patterns) with scatterplots (mapping performance metrics), enabling joint, interactive visual exploration of model structure, feature behavior, and predictive performance. Contribution/Results: VAR significantly enhances model selection efficiency and interpretability, facilitating rapid identification of optimal models across diverse application scenarios. It systematically reveals the internal diversity distribution within the Rashomon set and uncovers performance–structure trade-off patterns, thereby supporting principled, transparent model auditing and decision-making.

Technology Category

• Machine Learning: Ensemble Methods
• Computer Vision: Interpretability, Explainability, and Transparency
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Web data visualization
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Evaluating the performance of closely matched machine learning(ML) models under specific conditions has long been a focus of researchers in the field of machine learning. The Rashomon set is a collection of closely matched ML models, encompassing a wide range of models with similar accuracies but different structures. Traditionally, the analysis of these sets has focused on vertical structural analysis, which involves comparing the corresponding features at various levels within the ML models. However, there has been a lack of effective visualization methods for horizontally comparing multiple models with specific features. We propose the VAR visualization solution. VAR uses visualization to perform comparisons of ML models within the Rashomon set. This solution combines heatmaps and scatter plots to facilitate the comparison. With the help of VAR, ML model developers can identify the optimal model under specific conditions and better understand the Rashomon set's overall characteristics.

Problem

• Lack of visualization for comparing Rashomon set models horizontally
• Need to identify optimal ML models under specific conditions
• Understanding overall characteristics of Rashomon set models visually

Innovation

• Visual comparison of Rashomon set models
• Combines heatmaps and scatter plots
• Identifies optimal models under conditions