Solar Open 2 Technical Report

2026-07-22

AI Summary

This work addresses the challenges of context length, training efficiency, and cross-domain capability integration in long-horizon agent tasks by proposing a 250B-parameter sparse mixture-of-experts language model. The architecture employs a position-encoding-free hybrid attention mechanism to support million-token context windows, enhanced by a gated Delta rule that accommodates negative feature values, a value-based data filtering strategy, and a multi-teacher online policy distillation framework—collectively improving both training efficiency and agent performance. The model achieves state-of-the-art results on English benchmarks including MMLU-Pro, LiveCodeBench, and APEX-Agents, demonstrates top average performance on Korean benchmarks, and matches the capabilities of DeepSeek-V4-Pro on Ko-GDPval despite being less than one-sixth its size.

Abstract

We present Solar Open 2, a 250B-A15B Mixture-of-Experts language model built for long-horizon agentic tasks, scaled up from Solar Open 1 (Solar Open 100B). To hold entire agent trajectories in a single context, Solar Open 2 reaches a 1M-token window through a hybrid attention stack that interleaves one softmax layer among every three linear-attention layers, using no positional encoding and a gated delta rule extended to negative eigenvalues. To train at this scale under a fixed compute budget, we make training efficient in two ways: a stronger starting point, and higher-value data. For the starting point, we initialize Solar Open 2 from Solar Open 1, transferring the 5.69B-parameter shared skeleton that survives the architectural change and learning everything else through full pre-training. For the data, we curate for value per token: quality- and rarity-aware data curation and mixture-ratio optimization refine a 20T pool into a 10T mixture that, at equal token budget, outperforms the Solar Open 1 recipe. To build its agent skills, we train twelve domain specialists across purpose-built scenarios, then consolidate them into a single model by Multi-teacher On-Policy Distillation (MOPD). Against comparably sized open-weight models on English benchmarks, Solar Open 2 leads on MMLU-Pro, LiveCodeBench, and the APEX-Agents agentic suite, and stays competitive with the strongest (DeepSeek-V4-Flash and MiMo-V2.5) elsewhere. On Korean benchmarks, Solar Open 2 records the highest average of any model compared, including fast-tier closed APIs, and on Ko-GDPval, an in-house Korean officework-agent benchmark, it is competitive with DeepSeek-V4-Pro (1.6T) at less than a sixth of its size.

Problem

• long-horizon agentic tasks
• large language model
• efficient training
• agent trajectories
• compute budget

Innovation

• Mixture-of-Experts
• Hybrid Attention
• Multi-teacher On-Policy Distillation
• Long-context Modeling
• Data Curation