Reducing the Adaptation Gap Through Reachable Fisher Geometry

2026-09-28

AI Summary

This study addresses the inability of the Fisher trace scalar to accurately characterize local curvature and subspace gaps in parameter-efficient fine-tuning. To this end, it introduces the concept of "reachable Fisher geometry." Methodologically, the full-model Fisher information is defined via the pullback through the adapter Jacobian, while the reachable Fisher trace is computed using score gradient norms. These quantities are further quantified through Hessian-Fisher deficiency analysis and matrix-free Frobenius discrepancy estimation. The findings reveal that trace matching does not guarantee reduced operator discrepancies or improved matrix alignment. Moreover, although a higher Fisher trace correlates with greater subspace difficulty, it serves merely as a diagnostic heuristic rather than a mechanism for fully controlling local gap behavior.

Abstract

Parameter-efficient fine-tuning (PEFT) determines not only how many parameters are trained, but also which local directions a model can move in, so similar adapters can affect subgroup losses differently. Since curvature matrices are infeasible to form at adapter scale, scalar summaries such as the Fisher trace are often used instead. We study what the trace reveals and what it loses through the reachable Fisher: each subgroup's full-model Fisher pulled back through the adapter Jacobian. Under likelihood losses, it represents the Gauss-Newton curvature accessible to the adapter, and its trace can be computed from score-gradient norms without forming the full matrix. Under matched subgroup gradients, a positive-definite reachable-Fisher difference, with a margin exceeding the Hessian-Fisher defect, implies that every sufficiently small nonzero model-changing update increases the signed gap. In contrast, the restricted operator norm determines worst-case quadratic change, while a matrix-free Frobenius discrepancy bounds its reachable-Fisher component. Trace alone cannot certify definiteness or control matrix mismatch. Equal traces rule out a positive-definite difference but can still hide large operator discrepancies. Across 306 single-seed models, higher trace accompanies greater subgroup difficulty in 75.7 percent of 1,218 eligible evaluations, while trace matching reduces the best-worst subgroup gap in all 30 dataset-encoder-adapter combinations. However, held-out audits show that operator discrepancy decreases in 23 of 30 combinations, while the unbiased squared-Frobenius statistic decreases in only 16 of 30. Fisher trace is therefore a scalable diagnostic and training heuristic, but not a certificate of local gap behavior or matrix alignment.

Problem

• parameter-efficient fine-tuning
• adaptation gap
• reachable Fisher geometry
• subgroup loss
• Fisher trace

Innovation

• Reachable Fisher Geometry
• Parameter-Efficient Fine-Tuning
• Adaptation Gap
• Fisher Trace
• Matrix-Free Frobenius Discrepancy