TopoTuner: Topological Finetuning of Large Language Models

2026-07-18

AI Summary

This work addresses the high computational cost of full-parameter fine-tuning of large language models and the lack of structural guidance in existing parameter-efficient methods such as LoRA. The authors propose a topology-aware parameter-efficient fine-tuning framework that, for the first time, leverages persistent homology and Wasserstein distance to quantify topological changes in attention projection matrices during fine-tuning. This enables the construction of a transferable parameter-freezing strategy across tasks. Experiments on LLaMA, Mistral, and Qwen demonstrate that updating only 1–2% of parameters achieves performance on par with full fine-tuning and surpasses LoRA in seven out of nine settings. Moreover, the approach reduces average training time by 20.4% compared to full fine-tuning and by 5.5% relative to LoRA.

Abstract

Full fine-tuning remains a strong way to adapt pretrained LLMs, but it updates all weights and can be expensive. LoRA reduces the number of trainable parameters, but it does not directly answer which pretrained components should be trained and which can be frozen during adaptation. We introduce TopoTuner, a topology-guided fine-tuning framework for selective freezing of attention projection matrices. \method treats each projection matrix as a row cloud and uses Wasserstein distances between persistence diagrams to measure how its topology changes during fine-tuning. TopoTuner learns a reusable freezing profile from a source dataset and transfers it to efficiently fine-tune models on out-of-domain datasets, evaluating whether task-specific topological drift generalizes across question answering and sentiment analysis tasks. Across LLaMA-3.1-8B, Mistral-7B-v0.3, and Qwen3-8B-Base, TopoTuner is competitive with full fine-tuning while training only 1-2\% of the model parameters, and outperforms LoRA in 7 out of 9 model-dataset settings, which can change up to 39.57\% of the projection parameters. Along with minimized updates, TopoTuner reduces training time by 20.4\% relative to full fine-tuning and 5.5\% relative to LoRA on average. TopoTuner opens a new direction for reusable freezing profiles, where fine-tuning behavior learned on one dataset can be shared across multiple tasks.

Problem

• fine-tuning
• parameter efficiency
• model adaptation
• selective freezing
• large language models

Innovation

• topological fine-tuning
• parameter-efficient adaptation
• persistent homology
• freezing profile transfer
• Wasserstein distance