Bilinear Optimization Divergence: Diagnosing Factor-Constrained LoRA Continual Learning

2026-09-20

AI Summary

研究通过Bilinear Optimization Divergence诊断LoRA持续学习中的因子约束问题，提出Semi-Frozen Orthogonal Routing和Weight Residual Projection方法，有效减少了历史响应并改善了任务性能。

Abstract

Orthogonality in a LoRA factor does not by itself specify what the composed update protects: the answer depends on the task-start state, the parameterization, and the realized optimizer displacement. We formalize this question through Bilinear Optimization Divergence (BOD), an anchor-relative diagnostic of effective-update response on selected historical features. The finite-step analysis distinguishes two cases. In a shared adapter, protecting the routing displacement leaves a learned-anchor residual through the changing companion factor. In a fresh zero-output block, a feasible routing state can protect the composed update while both current factors remain trainable. These conditions yield Semi-Frozen Orthogonal Routing (SFOR) for shared adapters and current-block hard protection for cumulative O-LoRA; Weight Residual Projection (WRP) enforces the required displacement after the optimizer step. Controlled two-task traces verify the predicted residual paths, reducing normalized historical response from 19.12% to 0.005% in the shared family and from 7.72% to 0.002% in the cumulative family. Four-task experiments on Qwen3-8B characterize the resulting trade-offs: SFOR improves backward transfer (BWT) from -2.47 to -0.86 with nearly unchanged average accuracy (AA), while O-LoRA hard protection improves three-order mean AA from 80.27% to 81.30% and forgetting measure (FM) from 2.20 to 0.43. Component controls also show that stricter feasibility need not improve final task performance. Together, the analysis and evidence provide an architecture-conditioned account of which constraint to enforce, how to enforce it, and how to interpret its empirical value.

Problem

• Bilinear Optimization Divergence
• LoRA Continual Learning
• Orthogonality
• Effective-Update Response
• Factor-Constrained

Innovation

• Bilinear Optimization Divergence
• Semi-Frozen Orthogonal Routing
• Weight Residual Projection