Scaling Influence Functions in LLMs through Eigenbasis-Corrected One-Bit Gradient Projection

2026-09-29

AI Summary

This study addresses the prohibitive storage costs and scalability bottlenecks associated with full-gradient computation for influence functions in large language models. To overcome these limitations, this work proposes EOGP, a novel method that introduces the first feature-basis-corrected one-bit gradient projection mechanism. By integrating EK-FAC matrix factorization for dimensionality reduction, PCA compression, and 1-bit quantization, EOGP efficiently approximates optimal linear representations to estimate data influence. The primary contribution lies in preserving predictive accuracy on unseen queries under extremely constrained storage budgets. Experimental results demonstrate that EOGP surpasses baseline methods in prediction accuracy on GPT-2 while reducing storage requirements to merely one-sixteenth of the original cost. Furthermore, when applied to the OLMo model, it achieves competitive performance using only 1% of the standard storage footprint.

Abstract

Influence functions estimate how individual training examples affect the behavior of large language models (LLMs). Analyzing how training data influence different behaviors of an LLM involves repeated influence computation. Reusing stored training gradients reduces the computational cost, but storing full gradients is prohibitively expensive at LLM scale. We study how to compress these gradients while preserving influence estimates for future queries that are unknown at storage time. Through a worst-case analysis, we characterize the optimal fixed-dimensional linear representation and propose eigenbasis-corrected one-bit gradient projection (EOGP) to approximate it at scale. Specifically, EOGP uses EK-FAC to reduce gradient dimensionality, then applies PCA within the retained subspace to learn compression directions from the training gradients. We then apply one-bit quantization to the resulting coordinates, allowing more coordinates to be retained within a fixed storage budget. On GPT-2, EOGP predicts retraining outcomes more accurately than the evaluated compression baselines while using one-sixteenth of their per-example storage. On OLMo 2 SFT models from 1B to 32B parameters, EOGP remains competitive with the baselines allocated over 100 times as much storage per example.

Problem

• Influence Functions
• Large Language Models
• Gradient Compression
• Training Data Attribution

Innovation

• Influence Functions
• Gradient Compression
• One-Bit Quantization
• EK-FAC
• Large Language Models