On the Impact of the Utility in Semivalue-based Data Valuation

2025-02-10

AI Summary

This paper addresses the inconsistency and interpretability challenges in data valuation via the semi-value method, arising from arbitrary utility function selection—particularly in classification tasks—where it induces systematic bias. We identify the underlying geometric mechanism and introduce the novel concept of “spatial signatures,” establishing a geometric decoupling framework that disentangles utility functions from individual data point contributions. For the first time, this enables separable, two-dimensional embedded representations of utility and semi-value weights. Leveraging cooperative game theory and rigorous geometric analysis, we systematically characterize the sensitivity pathways of common classification utilities—including accuracy and arithmetic mean—to valuation outcomes. The framework provides both theoretically grounded, human-interpretable analysis tools and empirically enhanced stability and reliability in data valuation, thereby laying a principled foundation for trustworthy data value assessment.

Technology Category

• Reasoning under Uncertainty: Decision/Utility Theory
• Machine Learning: Calibration & Uncertainty Quantification
• Knowledge Representation and Reasoning: Preferences

Application Category

• Security and Privacy: Data transparency and provenance
• Semantics and Knowledge: Provenance, trust, security and privacy, and ethical issues in managing semantic data
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Semivalue-based data valuation in machine learning (ML) quantifies the contribution of individual data points to a downstream ML task by leveraging principles from cooperative game theory and the notion of utility. While this framework has been used in practice for assessing data quality, our experiments reveal inconsistent valuation outcomes across different utilities, albeit all related to ML performance. Beyond raising concerns about the reliability of data valuation, this inconsistency is challenging to interpret, as it stems from the complex interaction of the utility with data points and semivalue weights, which has barely been studied in prior work. In this paper, we take a first step toward clarifying the utility impact on semivalue-based data valuation. Specifically, we provide geometric interpretations of this impact for a broad family of classification utilities, which includes the accuracy and the arithmetic mean. We introduce the notion of spatial signatures: given a semivalue, data points can be embedded into a two-dimensional space, and utility functions map to the dual of this space. This geometric perspective separates the influence of the dataset and semivalue from that of the utility, providing a theoretical explanation for the experimentally observed sensitivity of valuation outcomes to the utility choice.

Problem

• Impact of utility on semivalue-based data valuation
• Inconsistency in valuation across different utilities
• Geometric interpretation of utility's influence on valuation

Innovation

• Semivalue-based data valuation
• Geometric interpretation of utility
• Spatial signatures in classification