Cross-Domain Hybrid OPD for Generalizable Search Agents

2026-08-03

AI Summary

This work addresses the "alignment tax" in reinforcement learning, where search agents suffer degraded general capabilities due to over-specialization. To mitigate this trade-off, the paper proposes a hybrid training strategy based on the Hunyuan3 architecture that treats specialization and general competence as complementary objectives. The approach integrates autonomous search-based reinforcement learning with cross-domain expert online policy distillation (OPD), enabling the agent to enhance its search proficiency while simultaneously recovering and strengthening its general-purpose abilities. Experimental results demonstrate that the proposed model maintains state-of-the-art search performance while significantly improving performance on general tasks, thereby achieving an effective balance between specialization and generalization.

Abstract

Recent advances in Reinforcement Learning (RL) have substantially improved the capabilities of autonomous search agents, enabling sophisticated planning, and iterative retrieval over dynamic information sources. However, optimizing language models for specialized search behaviors often incurs an alignment tax, where gains in search performance come at the expense of general-purpose capabilities, limiting their effectiveness as universal assistants. In this technical report, we present the training framework behind the Yuanbao search agent, designed to achieve search specialization without sacrificing general intelligence. Built upon the Hunyuan3 architecture, our framework combines agentic reinforcement learning for autonomous search with a cross-domain expert On-Policy Distillation (OPD) pipeline. Experts specializing in complementary general-purpose domains are distilled into the search-specialized student, restoring and further enhancing its broad capabilities. Rather than treating specialization and general capability as competing objectives, our hybrid training strategy jointly optimizes both, effectively mitigating the alignment tax. Extensive experiments demonstrate that the resulting model achieves competitive search performance while consistently improving its general-purpose capabilities, providing a favorable balance between specialized execution and broad generalization in real-world search scenarios.

Problem

• alignment tax
• search agents
• general-purpose capabilities
• specialization
• Reinforcement Learning

Innovation

• Cross-Domain Hybrid OPD
• On-Policy Distillation
• Reinforcement Learning
• Alignment Tax
• Generalizable Search Agents