A Tropical Geometry View of Forgetting: A Per-Unit Projector for Knowledge-Preserving Fine-Tuning

2026-10-03

AI Summary

This study addresses the knowledge forgetting problem in large language model fine-tuning caused by overly coarse shared subspace projections. We propose a gate-aware, per-unit projection method that leverages tropical geometry to analyze ReLU layer structures, revealing an exact duality between data and weight spaces. By integrating Zonotope polytope theory with the budget separation theorem, we theoretically demonstrate that per-unit projection significantly outperforms global shared subspaces in minimizing the cost of knowledge preservation. Experiments on OPT models show that our approach substantially reduces the forgetting rate at lower constraint dimensions, validating the critical role of open/closed partition mechanisms. Ultimately, this work achieves efficient, knowledge-preserving fine-tuning for large language models.

Abstract

Fine-tuning a language model on new text degrades what it already does. Replay-free projectors such as Adam-NSCL and GPM forbid one shared subspace of a layer's inputs in every row of the update. The tropical geometry of a ReLU layer shows why this is too coarse. In data space, the units'walls are tropical hypersurfaces whose cells are dual to the upper vertices of a zonotope; in weight space, each old token is a hyperplane, and the tokens cut out a polyhedron, the closure of the weights that keep every token on its side. An exact identity joins the two pictures: the squared change of the layer's output under any weight change splits into in-cell, open-to-closed and closed-to-open terms, and the first two live on the tokens each unit fires on. The identity names a gate-aware per-unit projector, and a budget-separation theorem prices exact protection: it costs a unit the rank of its own open tokens, while a shared subspace pays at least the rank of their union in every row. On OPT-1.3b, where 96% of (token, unit) pairs are closed, the projector forgets less than Adam-NSCL at all six matched budgets from 9 to 60 constrained directions per row, the gap widening from $1.1\times$ to $4.3\times$; with 1/5.5 of the directions it halves the forgetting of Adam-NSCL at GPM's energy threshold. On OPT-6.7b, it matches Adam-NSCL's forgetting at matched budget while learning more. As the theory predicts, the open/closed partition is the operative variable: open tokens beat random, sign-blind and anti-gate token sets on 18 of 18 seed-pairs. In pruning repair, every derivative-based local model of the output error at the dense weights is blind to pairs that open: the minimisers of the gate-weighted objective can leave the polyhedron, the objective's closed-form solution is 1.94 nats worse than no repair on OPT-1.3b, and a convex one-sided penalty bounds the escape.

Problem

• catastrophic forgetting
• knowledge-preserving fine-tuning
• language model fine-tuning
• replay-free projectors

Innovation

• Tropical Geometry
• Per-Unit Projector
• Knowledge-Preserving Fine-Tuning
• Budget-Separation Theorem
• Pruning Repair