EulerLoRA: Rank-Driven Jump Dynamics for Calibrated Parameter-Efficient Fine-Tuning

2026-08-02

AI Summary

Standard LoRA lacks the ability to estimate predictive uncertainty and struggles to produce diverse outputs. This work proposes EulerLoRA, which introduces a stochastic jump mechanism into the LoRA framework for the first time. By applying structured perturbations to the rank-one components of shared low-rank adapters, EulerLoRA generates multiple stochastic prediction trajectories with only a minimal increase in trainable parameters, while preserving a deterministic transformation in expectation. Evaluated on CIFAR-10, CIFAR-100, and HAM10000, the method achieves performance comparable to or better than LoRA ensembles, with approximately 69% fewer parameters, significantly enhancing prediction diversity, uncertainty calibration, and computational efficiency.

Abstract

Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning, but standard LoRA produces a single deterministic model and does not directly support predictive uncertainty estimation. We introduce EulerLoRA, a stochastic extension of LoRA that generates multiple predictive trajectories by sampling structured variations along the rank-one components of shared low-rank adapters, while preserving the deterministic LoRA transformation in expectation. We evaluate EulerLoRA with vision transformers on CIFAR-10, CIFAR-100, and HAM10000, together with out-of-distribution detection on SVHN. Across these benchmarks, EulerLoRA achieves comparable or improved performance relative to strong LoRA-Ensemble baselines. Using two rank-20 adapters, EulerLoRA requires approximately 3 million trainable adapter parameters, compared with about 10 million for a rank-8, 16-adapter LoRA-Ensemble, corresponding to roughly 69% fewer trainable parameters. These results show that useful predictive diversity can be obtained from a small number of shared adapters.

Problem

• Low-Rank Adaptation
• predictive uncertainty
• parameter-efficient fine-tuning
• stochastic modeling
• model calibration

Innovation

• EulerLoRA
• Low-Rank Adaptation
• Predictive Uncertainty
• Parameter-Efficient Fine-Tuning
• Stochastic Fine-Tuning