Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignment

2024-05-01

arXiv.org

AI Summary

Large language models (LLMs) face challenges in value alignment, while smaller models (e.g., 7B-parameter variants) exhibit weak safety robustness. Method: This paper proposes a self-alignment framework integrating structured reasoning chains with a Mixture-of-Experts (MoE) architecture. It introduces a four-stage safety reasoning chain—problem analysis, response guidance, safe generation, and safety verification—and pioneers a step-level routing multi-LoRA architecture enabling dynamic-length reasoning and lossless expert switching. Contribution/Results: Through self-alignment training and multi-LoRA fine-tuning, the framework significantly improves safety compliance, jailbreak resistance, and over-rejection mitigation. On a 7B model, it achieves alignment performance comparable to OpenAI’s o1, establishing a novel paradigm for efficient, lightweight model alignment. The approach bridges the gap between computational efficiency and rigorous safety assurance without sacrificing inference fidelity or expert specialization.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

As the capabilities of large language models (LLMs) continue to expand, aligning these models with human values remains a significant challenge. Recent studies show that reasoning abilities contribute significantly to model safety, while integrating Mixture-of-Experts (MoE) architectures can further enhance alignment. In this work, we propose Mixture of insighTful Experts (MoTE), a novel framework that synergistically combines reasoning chains and expert mixtures to improve self-alignments. From a data perspective, MoTE employs a structured reasoning chain comprising four key stages: Question Analysis, Answer Guidance, Safe Answer, and Safety Checking. This approach enhances safety through multi-step reasoning and proves effective even for smaller and less powerful LLMs (e.g., 7B models). From an architectural perspective, MoTE adopts a multi-LoRA framework with step-level routing, where each expert is dedicated to a specific reasoning step. This design eliminates the need for balance losses, ensures stable training, and supports adaptive inference lengths. Experimental results demonstrate that MoTE significantly improves model safety, jailbreak resistance, and over-refusal capabilities, achieving performance comparable to OpenAI's state-of-the-art o1 model.

Problem

• Aligning large language models with human values
• Enhancing model safety through reasoning chains
• Improving self-alignment using Mixture of Experts

Innovation

• Combines reasoning chains and expert mixtures
• Employs structured four-stage reasoning chain
• Adopts multi-LoRA framework with step-level routing