🤖 AI Summary
Current safety fine-tuning defenses are often validated by measuring the reduction in performance gaps on held-out sets; however, this metric is susceptible to sampling noise, topical artifacts, capability degradation, or non-transferable mechanisms, lacking a reliable evaluation standard. This work proposes the Acceptance Cards framework, which establishes—for the first time—a four-dimensional diagnostic criterion encompassing statistical reliability, novel semantic generalization, mechanistic alignment, and cross-task transferability, accompanied by an executable auditing toolkit for systematic validation of defense efficacy. Re-evaluating SafeLoRA on Gemma-2-2B-it across 46 experimental configurations reveals that it consistently fails to satisfy all four diagnostic criteria, exposing significant limitations in existing approaches.
📝 Abstract
Safe fine-tuning defenses are often endorsed on the basis of a held-out gap reduction, but the same reduction can come from sampling noise, subject artifacts, capability loss, or a mechanism that does not transfer. We introduce Acceptance Cards: an evaluation protocol, a documentation object, an executable audit package, and a claim-specific evidential standard for safe fine-tuning defense claims. The protocol checks statistical reliability, fresh semantic generalization, mechanism alignment, and cross-task transfer before treating a gap reduction as a full-card pass. Re-scored under this installed-gap protocol, SafeLoRA fails the full-card pass on Gemma-2-2B-it: under strict mechanism-class coding it fails all four diagnostics, and under a permissive shrinkage relabel it still fails three of four. This is a narrow installed-gap audit on one model family, not a global judgment of SafeLoRA's effectiveness. In a 46-cell audit, no cell satisfies the strict conjunction. The closest family is a near miss that passes reliability and mechanism checks where the required data are available, but fails the fresh-subject threshold, lacks a strict transfer pass, and carries a measurable deployment-accuracy cost.