Certification Frontiers for Gaussian LoRA: Independent Priors, Posterior Risk, and Prediction-Preserving Balancing

2026-09-26

AI Summary

This study addresses the limitation that calibrated Gaussian LoRA posteriors cannot automatically guarantee valid PAC-Bayes generalization bounds. To overcome this, the work delineates the Gaussian LoRA certification frontier by decoupling the interventions of priors, predictors, and complexity counting. It introduces a precise isotropic KL envelope to exclude overly diffuse posteriors and exploits the full GL(r) symmetry of low-rank factors to minimize the KL divergence in closed form, achieving prediction-preserving matrix balancing. Empirically, the proposed approach successfully certifies risks below 0.1 via Chernoff bounds across 20 datasets. Furthermore, matrix balancing yields an average 29.3% reduction in KL divergence without compromising predictive performance.

Abstract

Post-hoc Bayesian fine-tuning places Gaussians around trained low-rank adapters, yet a calibrated posterior does not by itself yield a useful generalization certificate. Such a posterior admits an informative PAC-Bayes certificate only when both the loss of its sampled predictors and its KL divergence from an admissible prior are small. In this research, we characterize this certification frontier for Gaussian LoRA posteriors and separate three interventions: changing the prior, changing the stochastic predictor, and changing only how its complexity is counted. First, an exact isotropic KL envelope eliminates the prior scale and yields a width threshold that excludes posterior widths before sampling, while a zero-KL floor identifies targets that no complexity reduction can reach at a measured risk bound. Second, we minimize KL in closed form over the full $GL(r)$ symmetry of the low-rank factors, leaving every sampled adapter product unchanged, and derive the noncentral objective required when the prior center is trained on an independent split. On a small-pool RoBERTa audit of 567 configurations, the recorded 64-draw summaries imply a certificate floor of $0.7298$ even with zero KL and Chernoff accounting, so reducing complexity alone cannot certify these posteriors at the recorded budgets. For a stable posterior in a controlled Gaussian-factor task, Chernoff accounting certifies risk below $0.1$ on 20 of 20 datasets with 1024 draws, whereas Hoeffding certifies none. On deliberately deformed synthetic rank-four factors, matrix balancing reduces KL by $29.3\%$ on average beyond scalar balancing without changing any prediction. Numerical split-prior scenarios make the remaining risk and complexity budgets explicit.

Problem

• PAC-Bayes certification
• Gaussian LoRA
• posterior risk bound
• KL divergence
• generalization certificate

Innovation

• PAC-Bayes certification
• Gaussian LoRA
• KL minimization
• prediction-preserving balancing
• Chernoff bound