HeuFouFT: Task-Guided Metaheuristic Coordinate Search for Fourier Fine-Tuning

2026-10-05

AI Summary

This study addresses the limitation of existing Fourier-based fine-tuning methods, which rely on fixed rules for spectral allocation and lack task adaptability. To overcome this, we propose HeuFouFT, a framework that introduces a novel task-guided metaheuristic search mechanism for frequency coordinates. Specifically, it employs random forests to filter candidate sets and integrates genetic simulated annealing, particle swarm optimization, and cuckoo search algorithms with lightweight probes and proxy tuning to achieve optimal spectral resource allocation. Evaluated on GPT-2, HeuFouFT surpasses baselines such as LoRA across multiple metrics while reducing trainable parameters by 37.6%. Furthermore, its computational overhead constitutes only 15%–18% of that required by full fine-tuning, demonstrating superior efficiency and effectiveness for parameter-efficient adaptation.

Abstract

We introduce Heuristic-Guided Fourier Fine-Tuning (HeuFouFT), a task-guided framework for selecting trainable frequency coordinates in Fourier fine-tuning. Existing uniform and Gaussian band-pass schemes allocate a limited spectral budget through fixed, task-agnostic rules. HeuFouFT instead searches for coordinates using downstream performance. A coarse intensity map from lightweight block-level probes initializes three metaheuristic optimizers: Genetic Algorithm with Simulated Annealing (GA-SA), Particle Swarm Optimization (PSO), and Cuckoo Search (CS). During search, a Random Forest filters each population so that only the top 30% of candidates proceed to proxy fine-tuning. On E2E with GPT-2-Medium, all three variants outperform random-uniform FourierFT, Gaussian band-pass FourierFT, and LoRA across five metrics. PSO further outperforms LoCA, the best-performing baseline, on four metrics while using 37.6% fewer trainable spectral coefficients. Once coordinates are selected, HeuFouFT requires only 15--18% FLOPs of Full FT. These results show that task-guided search allocates limited spectral capacity more effectively than fixed sampling. Our code is publicly available.

Problem

• Fourier fine-tuning
• spectral budget allocation
• task-agnostic rules
• frequency coordinate selection

Innovation

• Fourier Fine-Tuning
• Metaheuristic Optimization
• Task-Guided Search
• Random Forest Filtering
• Parameter-Efficient Fine-Tuning