Fisher-Guided Progressive Parameter Selection for Adaptive Fine-Tuning

2026-06-08

AI Summary

This work addresses the limitation of existing parameter-efficient fine-tuning methods, which rely on fixed architectural heuristics to select trainable parameters and lack task adaptivity. The authors propose a dynamic parameter selection mechanism grounded in PAC-Bayesian theory, which tracks the temporal geometric drift of the Fisher information matrix. By measuring the discrepancy between consecutive Fisher distributions using a scale-invariant Jensen–Shannon divergence, the method adaptively freezes parameters whose curvature contributions remain stable, progressively constructing a task-aware, efficient trainable subset. Integrating Fisher information, generalization bound decomposition, and parameter-efficient fine-tuning, this approach significantly enhances both in-distribution performance and zero-shot transfer capabilities on downstream segmentation tasks, thereby validating the efficacy of Fisher structure drift as an adaptive signal.

Abstract

Parameter-efficient fine-tuning (PEFT) aims to adapt pretrained models with a small trainable parameter subset, however, most existing methods choose this subset from fixed architectural heuristics rather than using dynamic, task-aware criteria. We introduce \textbf{FisherAdapTune}, a Fisher-guided Adaptive Fine-Tuning framework that progressively selects parameter groups by tracking the temporal drift of their Fisher geometry. Starting from a PAC-Bayesian view of fine-tuning, we decompose the generalization error bound into Fisher-weighted update costs and show that parameter groups whose curvature contribution has stabilized can be frozen to reduce the error bound without interrupting the remaining adaptation dynamics. FisherAdapTune formulates this criterion with a scale-invariant Jensen-Shannon distance between consecutive Fisher distributions, yielding an adaptive active parameter set. We evaluate our approach on a downstream segmentation task, and results show FisherAdapTune improves the in-distribution performance and zero-shot transfer in multiple settings, validating that Fisher structural drift is a useful signal for efficient, task-aware adaptation. We release our \href{https://github.com/AtlasAnalyticsLab/FisherAdapTune}{code} publicly to enable further application of our proposed approach.

Problem

• parameter-efficient fine-tuning
• task-aware adaptation
• Fisher information
• adaptive parameter selection
• generalization error

Innovation

• Fisher information
• parameter-efficient fine-tuning
• adaptive selection
• PAC-Bayesian
• Jensen-Shannon distance