Token-level Accept or Reject: A Micro Alignment Approach for Large Language Models

2025-05-26

AI Summary

To address the high computational cost and poor transferability of conventional alignment methods (e.g., RLHF, DPO) in aligning large language models (LLMs) with human preferences, this paper proposes MARA—a novel token-level fine-grained alignment paradigm. MARA decouples sentence-level preference modeling into lightweight token-level binary classification tasks, employing a compact, LLM-independent three-layer fully connected network to directly predict whether each token should be accepted or rejected—eliminating the need for LLM parameter fine-tuning. This design achieves strict parameter decoupling between the alignment module and the backbone model, substantially improving cross-model plug-and-play capability and training efficiency. Evaluated on seven mainstream LLMs and three open-source preference datasets, MARA consistently outperforms baseline methods in alignment performance while reducing training costs by over an order of magnitude. The code and models are publicly released.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Humans and AI: Learning Human Values and Preferences

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

With the rapid development of Large Language Models (LLMs), aligning these models with human preferences and values is critical to ensuring ethical and safe applications. However, existing alignment techniques such as RLHF or DPO often require direct fine-tuning on LLMs with billions of parameters, resulting in substantial computational costs and inefficiencies. To address this, we propose Micro token-level Accept-Reject Aligning (MARA) approach designed to operate independently of the language models. MARA simplifies the alignment process by decomposing sentence-level preference learning into token-level binary classification, where a compact three-layer fully-connected network determines whether candidate tokens are"Accepted"or"Rejected"as part of the response. Extensive experiments across seven different LLMs and three open-source datasets show that MARA achieves significant improvements in alignment performance while reducing computational costs. The source code and implementation details are publicly available at https://github.com/IAAR-Shanghai/MARA, and the trained models are released at https://huggingface.co/IAAR-Shanghai/MARA_AGENTS.

Problem

• Aligning LLMs with human preferences efficiently
• Reducing computational costs of existing alignment techniques
• Simplifying alignment via token-level binary classification

Innovation

• Token-level binary classification for alignment
• Compact three-layer network for efficiency
• Operates independently of language models