OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling

2026-08-05

AI Summary

Existing long-context corpora lack cross-repository long-range dependencies, limiting the performance of language models on long-context tasks. This work proposes OctoLong—the first systematic pipeline for constructing ultra-long contexts by integrating cross-repository code dependency structures. It leverages abstract syntax tree (AST) parsing, language servers, and recursive package manager queries to extract inter-repository reference relationships, which are then injected into models via a two-stage paradigm comprising intermediate pretraining and instruction fine-tuning. Remarkably, using only 12% of the OctoLong data yields significant performance gains across 18 prominent open-source long-context models on tasks involving long-range retrieval, state tracking, code comprehension, and agent reasoning, thereby demonstrating the critical role of cross-repository dependencies in effective long-context modeling.

Abstract

Context lengths of language models (LMs) have dramatically increased, driven by the demands for in-context learning, self-improvement, and long-horizon agentic workflows. Existing long-context corpora, however, are dominated by books, academic articles, and code repositories, which are finite resources and often scarce in long-distance dependencies. In this work, we introduce OctoLong, a context engineering pipeline that instruments an AST parser, a language server backend, and a package manager to facilitate the recursive retrieval of code references, enabling the curation of dependency-rich code contexts of millions of tokens in length. We then train OctoLong-Instruct, a suite of capable long-context open LMs, derived from base models ranging in size from 600M to 14B parameters, via context-extension mid-training on a ~50B-token mixture containing ~6.2B tokens of OctoLong code contexts, followed by ~10B tokens of instruction tuning. Our training ablations and experimental evaluations against 18 state-of-the-art open-weight long-context LMs show that supplanting just 12% of traditional context-extension corpora with OctoLong data yields substantial gains in long-range retrieval, long-term state tracking, repository-level code understanding, and downstream agentic tasks, while also enhancing API usage in short-context coding scenarios.

Problem

• long-context modeling
• code repositories
• long-distance dependencies
• language models
• context extension

Innovation

• long-context modeling
• cross-repository code
• mid-training
• dependency-rich context
• code understanding