Are Parameter-Efficient Fine-tuning Methods Really Different?

2026-10-06

AI Summary

This study addresses the unclear functional and geometric distinctions among parameter-efficient fine-tuning (PEFT) methods and the necessity of spectrum preservation. By comparing six PEFT approaches, including orthogonal fine-tuning and LoRA variants, through spectral analysis and intervention experiments, it systematically evaluates their effects on task performance, catastrophic forgetting, and the geometric structure of pretrained weights. The findings reveal that explicit geometric preservation is not a necessary condition for performance guarantees, proposing a new evaluation paradigm based on functional consequences rather than mere preservation. Experiments demonstrate that LoRA most effectively mitigates catastrophic forgetting, while DoRA achieves superior overall performance. Furthermore, restoring dominant spectral components exerts the most significant impact on model performance.

Abstract

Parameter-efficient fine-tuning (PEFT) offers many parameterizations, yet their methodological and functional differences remain unclear. We compare six methods in language and diffusion models to examine how their parameterizations relate to task performance, forgetting, and changes in pretrained weight geometry. Motivated by the spectrum-preserving design of orthogonal fine-tuning (OFT), we first ask whether spectral preservation is itself important for adaptation and retention. We find that the selected LoRA-family methods also approximately preserve pretrained geometry, and that restoring their slightly drifted singular-value spectra largely preserves task performance, questioning the necessity of explicit geometric preservation. Beyond this, we observe that some methods exhibit distinct adaptation--retention trade-offs that vary across settings: LoRA most consistently limits forgetting at competitive performance, DoRA achieves higher mean task scores than LoRA in most comparisons, while PiSSA often incurs greater retention costs. Further intervention experiments suggest that while performance gains from different PEFT methods can be attributed to modifications in different groups of spectral components, we consistently find that restoring dominant rather than intermediate or trailing components produces the largest mean reduction in general-text NLL or base-image drift. Together, these results motivate evaluating geometric constraints through their functional consequences rather than preservation alone. Code is available at https://github.com/Kuaaannn/PEFT_methods.

Problem

• Parameter-Efficient Fine-tuning
• Spectral Preservation
• Catastrophic Forgetting
• Weight Geometry
• Adaptation-Retention Trade-off

Innovation

• Parameter-Efficient Fine-Tuning
• Spectral Preservation
• Weight Geometry
• Catastrophic Forgetting
• Singular Value Spectrum