Stream Aligner: Efficient Sentence-Level Alignment via Distribution Induction

2025-01-09

AI Summary

Large language models (LLMs) suffer from poor cross-task adaptability, low real-time responsiveness, and heavy reliance on auxiliary large models in value alignment. To address these challenges, we propose the Streaming Distribution Induce Aligner (Stream Aligner), the first framework to introduce *streaming distribution-induced alignment*—a paradigm enabling sentence-level, low-latency, embedded dynamic alignment during LLM autoregressive generation. Stream Aligner employs a lightweight preference model to infer user intent and iteratively refines suffix tokens without invoking external large models. Experiments demonstrate that Stream Aligner-2B improves helpfulness by 76.1% and harmlessness by 36.0% for Llama2-70B-chat; Stream Aligner-8B enhances mathematical reasoning performance by 3.5% for Llama3-70B-Instruct. Our approach significantly advances alignment efficiency and generalization across diverse tasks and model scales.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Multiagent Systems: Adversarial Agents

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The rapid advancement of large language models (LLMs) has led to significant improvements in their capabilities, but also to increased concerns about their alignment with human values and intentions. Current alignment strategies, including adaptive training and inference-time methods, have demonstrated potential in this area. However, these approaches still struggle to balance deployment complexity and capability across various tasks and difficulties. In this work, we introduce the Streaming Distribution Induce Aligner (Stream Aligner), a novel alignment paradigm that combines efficiency with enhanced performance in various tasks throughout the generation process. Stream Aligner achieves dynamic sentence-level correction by using a small model to learn the preferences of the suffix sentence, iteratively correcting the suffix sentence output by the upstream model, and then using the corrected sentence to replace the suffix sentence in subsequent generations. Compared to Aligner, our experiments demonstrate that Stream Aligner reduces reliance on the capabilities of additional models, enhances the reasoning abilities of LLMs, and decreases latency during user interaction. Specifically, Stream Aligner-2B model has achieved an improvement of 76.1% in helpfulness, 36.0% in harmlessness on the tested Llama2-70B-chat model, and Stream Aligner-8B has achieved an improvement of 3.5% on the math ability of the tested Llama3-70B-Instruct model.

Problem

• Large Language Models
• Human Values Alignment
• Task Adaptability and Efficiency

Innovation

• Stream Aligner
• Language Model Efficiency
• Human Value Alignment