Faithful and Fast Influence Function via Advanced Sampling

2025-10-30

AI Summary

To address the high computational cost of Hessian matrix estimation and inconsistent IF approximations caused by random sampling in black-box influence function (IF) computation, this paper proposes a dual-path representative subset sampling method leveraging both feature-space geometry and logits distribution. Without requiring access to model parameters or internal gradients, the method significantly reduces both the variance and computational complexity of IF estimation. It employs gradient- and Hessian-informed sampling strategies to jointly ensure statistical representativeness and computational tractability. Experiments demonstrate that, compared to baseline approaches, the proposed method reduces computation time by 30.1%, decreases memory consumption by 42.2%, and improves F1-score by 2.5%. Moreover, it achieves superior explanation consistency and practical utility in downstream tasks such as model diagnosis and machine unlearning.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Sampling/Simulation-based Search
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

How can we explain the influence of training data on black-box models? Influence functions (IFs) offer a post-hoc solution by utilizing gradients and Hessians. However, computing the Hessian for an entire dataset is resource-intensive, necessitating a feasible alternative. A common approach involves randomly sampling a small subset of the training data, but this method often results in highly inconsistent IF estimates due to the high variance in sample configurations. To address this, we propose two advanced sampling techniques based on features and logits. These samplers select a small yet representative subset of the entire dataset by considering the stochastic distribution of features or logits, thereby enhancing the accuracy of IF estimations. We validate our approach through class removal experiments, a typical application of IFs, using the F1-score to measure how effectively the model forgets the removed class while maintaining inference consistency on the remaining classes. Our method reduces computation time by 30.1% and memory usage by 42.2%, or improves the F1-score by 2.5% compared to the baseline.

Problem

• Explaining training data influence on black-box models using influence functions
• Reducing computational cost of influence functions through advanced sampling
• Improving accuracy and consistency of influence function estimations

Innovation

• Sampling based on features and logits
• Selecting representative subset for influence functions
• Reducing computation time and memory usage