Instruction-Conditioned Exploration with Asymmetric Reinforcement Learning and Self-Distillation

2026-08-03

AI Summary

This work addresses the challenge that large language models often struggle to effectively leverage pre-trained knowledge for generating diverse behaviors in reinforcement learning due to complex action spaces and insufficient exploration. To overcome this, the authors propose Instruction-Conditioned Exploration (ICE), a mechanism that enriches behavioral diversity during training by augmenting task prompts with varied instructions. ICE integrates asymmetric reinforcement learning with a self-distillation objective to transfer these exploratory behaviors to an unconditional policy at test time. Evaluated on the Qwen3-1.7B model, the method achieves a 5.0% improvement over DAPO in held-out pass@1 performance on mathematical reasoning tasks with 4K response lengths and maintains this advantage in 8K long-context settings, significantly enhancing the model’s generalization capability.

Abstract

Post-training Large Language Models (LLMs) with Reinforcement Learning (RL) has become an important tool for improving model capabilities, but the LLM action-space structure introduces challenges distinct from classical RL, with implications for inducing exploration. New methods are required that leverage the broad knowledge and flexibility of pre-trained LLMs to deliberately generate diverse experience at training time. We propose Instruction-Conditioned Exploration (ICE), which supplements task prompts during training with one of several distinct instructions, increasing the coverage of behaviours attempted. To facilitate ICE, we propose Asymmetric-RL/SD, a combined Reinforcement Learning and Self-Distillation training objective, to transfer explored behaviours to the unconditioned test-time policy. ICE with the Asymmetric-RL/SD objective improves Qwen3-1.7B held-out pass@1 performance at $4$K response length on mathematical reasoning tasks by $5.0\%$ relative to training with DAPO, with improvement persisting at a longer 8K context.

Problem

• Instruction-Conditioned Exploration
• Reinforcement Learning
• Large Language Models
• Exploration
• Post-training

Innovation

• Instruction-Conditioned Exploration
• Asymmetric Reinforcement Learning
• Self-Distillation
• LLM Post-training
• Exploration in RL