Ready2Blend: From Natural-Language Instructions to Composable Alignment Prompts

2026-09-30

AI Summary

This study addresses the challenges of weak natural language instruction controllability, post-training forgetting, and high computational costs in the continuous alignment of large language models by proposing the Ready2Blend framework. This method introduces AlignFormer to map natural language requirements into modular prompts, leveraging compositional regularization to enable flexible blending and reweighting at inference time with a frozen backbone network. By combining the flexibility of natural language with the precision of learned alignment, the framework supports retraining-free personalization and order-agnostic composition. Experimental results demonstrate that the proposed approach achieves 93.1%–98.5% of the performance upper bound established by joint training while reducing training time by 4.3×.

Abstract

Continual alignment requires LLMs to adapt to new requirements without forgetting previously acquired behaviors. Natural-language instructions are flexible and composable but offer only indirect control, whereas post-training provides stronger adaptation at the cost of repeated parameter updates. We introduce Ready2Blend, which combines the flexibility of natural language with learned alignment. AlignFormer maps each requirement to a fixed-length alignment prompt stored in a modular prompt bank, while the backbone and prior prompts remain frozen. Composability regularization transfers the semantic geometry of textual requirements into prompt space, enabling inference-time blending and reweighting. Across two practical continual alignment settings, Ready2Blend is the only frozen-backbone method that matches post-training-based alignment methods, reaching $93.1$-$98.5\%$ of a joint-training reference with competitive retention, while requiring only a few prompt tokens and up to $4.3\times$ less training time. Its modular design further enables weighted personalization and order-free composition without retraining. Code will be released upon acceptance.

Problem

• Continual Alignment
• Large Language Models
• Composability
• Parameter-efficient Adaptation
• Natural-language Instructions

Innovation

• Continual Alignment
• Alignment Prompts
• Composability Regularization
• Modular Prompt Bank
• AlignFormer