The Blessing of Dimensionality: How Near-Orthogonality in High-Dimensional Spaces Explains Temporal Portability

2026-07-22

AI Summary

This study addresses the lack of theoretical grounding for the long-term temporal portability of PortLLM during continual pretraining. Through an empirical investigation involving ten rounds of continual pretraining on Mistral, Gemma, and Qwen models, combined with high-dimensional near-orthogonality analysis, the work reveals the mechanism by which PortLLM maintains performance without repeated fine-tuning. Innovatively adopting a high-dimensional geometric perspective, the paper identifies near-orthogonality in parameter update directions as the key theoretical basis for PortLLM’s temporal portability. The validity of this insight is further corroborated through comparisons with LoRA and analyses of loss landscape geometry. Results demonstrate that PortLLM sustains strong performance across multiple model updates, substantially reducing the fine-tuning overhead typically associated with continual learning.

Abstract

Fine-tuning has been widely used to adapt large language models (LLMs) for domain-specific tasks. Parameter efficient fine-tuning (PEFT) methods such as low-rank adaptation (LoRA) are frequently used to reduce computational costs. PortLLM is a training-free and data-free scheme used to adapt LLMs after continual pretraining. Although the initial PortLLM results show that LoRA patches exhibit short-term temporal portability, the long-term performance of PortLLM across several updates of continual pretraining remains underexplored. Furthermore, the intriguing effectiveness of PortLLM is not well understood from a theoretical standpoint. We address these two open questions by (1) performing an extensive empirical study of the long-term temporal portability of PortLLM patches across 10 continual pretraining steps using base models Mistral, Gemma, and Qwen; and (2) offering two theoretical analyses to explain our observation that the simple PortLLM method achieves competitive performance. We find empirically that the portability persists across longer time duration, indicating that repeated fine-tuning is not required when the base model is periodically updated. We find theoretically that near-orthogonality of high-dimensional vectors is a key justification for temporal portability. Our analyses also demonstrate a geometric perspective of the loss landscape in facilitating the theoretical comparison of different adaptation options.

Problem

• temporal portability
• continual pretraining
• PortLLM
• high-dimensional spaces
• parameter-efficient fine-tuning

Innovation

• temporal portability
• near-orthogonality
• high-dimensional geometry
• parameter-efficient fine-tuning
• PortLLM