ALIGNBEAM : Inference-Time Alignment Transfer via Cross-Vocabulary Logit Mixing

2026-06-10

AI Summary

This work addresses the challenge that domain-specific fine-tuning often compromises the safety of large language models (LLMs), while existing inference-time defense mechanisms struggle to generalize across model families due to vocabulary mismatches. To overcome this, the authors propose a training-free, inference-time alignment transfer method that maps logits from a secure anchor model to a target model through cross-vocabulary logit projection. The approach generates K candidate continuations via beam search and selects the safest output using a lightweight LLM-based judge. This method achieves, for the first time, cross-model-family safety alignment without modifying model weights, enabling flexible trade-offs between safety and utility at deployment time. It significantly improves refusal rates on adversarial benchmarks while preserving task accuracy and maintaining manageable inference overhead.

Abstract

Domain fine-tuning degrades the safety of large language models: fine-tuned specialists readily comply with harmful prompts framed in domain language. Existing inference-time defenses that mix logits from a safe anchor model require both models to share a vocabulary, which rules them out for the cross-family specialists where safety is most degraded. We present ALIGNBEAM, a training-free method that lifts this restriction by translating anchor logits into the target model's vocabulary token-by-token at each decoding step; a small LLM judge then selects the safest among K candidate continuations. No weights are changed, and the safety-utility trade-off can be tuned at deployment without retraining. Across both cross-vocabulary and same-vocabulary evaluation pairs, ALIGNBEAM substantially raises refusal on adversarial benchmarks while keeping task accuracy and inference overhead within practical bounds. The results show that safety alignment can be transferred between model families at inference time, without touching either model's weights.

Problem

• safety alignment
• cross-vocabulary
• domain fine-tuning
• harmful prompts
• large language models

Innovation

• inference-time alignment
• cross-vocabulary logit mixing
• training-free safety
• model-agnostic defense
• safety alignment transfer