CritLens: Visual Analytics for Criteria Discovery in Review-Based Decision Making

2026-06-06

AI Summary

This study addresses the challenge users face in constructing, refining, and validating personalized evaluation criteria when making decisions based on reviews, particularly due to the frequent oversight of infrequent yet critical details. To tackle this issue, the authors propose a visual analytics system that integrates large language models with the Analytic Hierarchy Process (AHP) to automatically generate an initial decision model from user reviews and support human-AI collaborative iterative refinement of criteria, weights, and evidence provenance. The system introduces a novel coverage gap detection mechanism to identify missing evaluation dimensions, incorporates AHP-based consistency constraints for interactive weight adjustment, and enhances decision transparency through multi-level scorecards and exportable reports. Experimental results demonstrate that the system significantly improves users’ control over their evaluation criteria and the overall trustworthiness of their decisions.

Abstract

We present CritLens, a visual analytics system that helps users build personalized multi-criteria decision models from review text. In everyday decisions -- choosing equipment, hotels, or restaurants -- evaluation criteria are either preset by platforms or generated by LLMs, leaving users unable to discover, adjust, or verify them against the underlying evidence. This is problematic because many preferences are latent: they surface only upon encountering specific reviews, and any fixed framework risks overlooking low-frequency but decisive details. CritLens addresses this gap by using LLMs to transform reviews into an initial AHP decision model, then supporting iterative, human-in-the-loop refinement. Through coverage gap detection in the embedding space, users discover criteria missed by the initial model; through interactive weight adjustment under AHP consistency constraints, they express personal priorities; and through a multi-level scorecard and exportable decision report, they trace every ranking back to the original review text. Two case studies, an eight-participant user study, and a quantitative consistency-repair experiment demonstrate the system's effectiveness.

Problem

• criteria discovery
• review-based decision making
• multi-criteria decision models
• latent preferences
• evaluation criteria

Innovation

• visual analytics
• criteria discovery
• AHP
• human-in-the-loop
• LLM