One Model, Multiple Goals: Adaptive Multi-Objective Learning for E-commerce Dialogue Systems

2026-06-08

AI Summary

This work addresses the training instability in e-commerce dialogue systems caused by misalignment between user state inference and natural language generation objectives. The authors propose MORE, a novel framework that formulates reasoning as a constrained policy optimization guided by explicit constraints and directly generates responses during inference without additional computational overhead. To further enhance performance, MORE incorporates an adaptive multi-reward mechanism that dynamically balances linguistic quality metrics. Integrating constrained reinforcement learning, gradient-driven reward weighting, and end-to-end response generation, the method significantly outperforms strong baselines on both ByteDance’s real-world system and MultiWOZ 2.2. In a 14-day online A/B test, MORE achieved a 16.53% increase in overall conversion rate, a 30.09% improvement in reach-to-conversion rate, higher user satisfaction, and reduced human-agent handoff rates—delivering 60% of the incremental gain observed from human agents.

Abstract

Dialogue systems in e-commerce scenarios often need to satisfy multiple objectives: accurately reasoning over user profiles (e.g., eligibility, credit limit) to ensure correct decision-making and user state interpretation, while also generating natural and faithful responses. These goals are complementary but not identical. In this work, we propose MORE, an adaptive Multi-Objective REinforcement learning framework that jointly optimizes reasoning accuracy and linguistic naturalness. Our preliminary experiments show that directly mixing rewards with diverging optimization dynamics can cause oscillations and unstable learning. Thus, instead of optimizing a single mixed reward, we treat reasoning functions as constraints that guide policy optimization. At inference time, the system directly generates responses without explicit reasoning steps, while still benefiting from reasoning-enhanced scaffold and avoiding additional inference overhead. To better balance linguistic objectives during response generation, we introduce an adaptive multi-reward mechanism that aggregates signals such as fluency and naturalness and dynamically reweighs them via gradient feedback. We evaluate MORE on two real-world dialogue systems at ByteDance and the MultiWOZ 2.2 benchmark, where it consistently outperforms strong baselines. In 14-day online experiments on ByteDance production traffic, MORE improves overall and reached conversion by 16.53% and 30.09%, while increasing user satisfaction and reducing handoff rates. Notably, in a human-machine comparison, MORE recovers about 60% of the incremental conversion lift achieved by human agents.

Problem

• multi-objective learning
• e-commerce dialogue systems
• reasoning accuracy
• linguistic naturalness
• reward balancing

Innovation

• Multi-Objective Reinforcement Learning
• Constrained Policy Optimization
• Adaptive Reward Aggregation
• E-commerce Dialogue Systems
• Reasoning-Aware Generation