codebook design

This work addresses the limitations of conventional tokenizers—such as Byte Pair Encoding (BPE)—which often misalign with linguistic structure, exacerbate biases, and inefficiently consume model capacity in multilingual and multidomain settings, while lacking systematic design and evaluation protocols. Treating tokenization as a core modeling decision for large language models, this study proposes the first context-aware framework that co-designs tokenization with the model architecture, integrating linguistic knowledge, domain-specific characteristics, and deployment constraints. By jointly optimizing the tokenizer and the model, and establishing standardized evaluation benchmarks alongside transparent reporting practices, the research provides both theoretical foundations and practical pathways toward more equitable, efficient, and adaptable language technologies, significantly enhancing performance and robustness across diverse languages and domains.

Traditional software engineering practices are human-centric and ill-suited to the requirements of large language model (LLM) agents in code understanding and generation. This work proposes the principle of semantic density optimization, advocating for the decoupling of semantic intent from human-readable representations and introducing novel code representations such as program skeletons. Through controlled experiments, the study evaluates the impact of four log formats—human-readable, structured, compressed, and tool-assisted compressed—on agent performance. Results show that excessive compression, while reducing input tokens by 17%, increases reasoning overhead and raises total conversation cost by 67%, thereby underscoring the critical importance of preserving tokens with high semantic value.

Byte Pair Encoding (BPE) tokenization yields subword sequences that lack direct support for formal language operations, hindering rigorous pattern matching and compositional verification in open-vocabulary NLP systems. Method: We propose the first deterministic finite automaton (DFA) construction algorithm tailored to BPE output—treating tokenized sequences as constrained symbol strings without reconstructing original bytes or characters. Our approach introduces a novel equivalence-class partitioning scheme and transition function synthesis mechanism grounded in BPE merge rules, enabling linear-time O(n) DFA construction while preserving semantic fidelity. Contribution/Results: The resulting DFA supports efficient subword-level regular expression matching, lexicon equivalence checking, and formal language composition operations. It significantly improves both efficiency and composability of pattern recognition and formal verification in open-vocabulary NLP, establishing foundational automata infrastructure for verifiable, scalable, tokenization-aware language processing.

This study addresses the limitations of large language model (LLM)-based multi-agent systems in software engineering, particularly the lack of transparency in resource consumption, unpredictability of costs, and unclear environmental impact. To this end, it introduces the first standardized token consumption evaluation framework tailored for agent-based software engineering. By analyzing execution trajectories from the ChatDev framework across 30 development tasks, the work maps internal agent interactions to standard software engineering phases—design, coding, completion, code review, testing, and documentation—and quantifies the distribution of input, output, and reasoning tokens across these stages. The analysis reveals that the code review phase alone accounts for 59.4% of total token usage, with input tokens comprising 53.9% of the total, indicating that cost is primarily driven by automated refinement and validation rather than initial code generation. These findings provide empirical foundations for optimizing workflows, forecasting costs, and designing efficient agent collaboration protocols.

This study addresses the lack of systematic empirical evidence regarding the real-world impact of coding agents in software development. Leveraging Mining Software Repositories (MSR) methods, it presents the first large-scale analysis of activity traces from large language model–based coding agents on GitHub, systematically identifying their behavioral patterns, potential risks, and effective usage strategies in authentic development environments. The research yields a set of empirically grounded insights concerning optimal timing for agent adoption, reliability concerns, and practical heuristics for deployment. These findings fill a critical gap in the literature, offering actionable guidance for developers and establishing a foundation for future investigations into AI-assisted programming.

This work addresses the inefficiency of large language models in code generation, where lengthy natural language constraint prompts incur high token costs without consistently improving constraint adherence. The authors propose replacing conventional prompts with compact, structured constraint headers and conduct the first systematic, multi-model, multi-task study to disentangle the effects of token reduction from constraint compliance. Through multi-round controlled experiments, rule-based scoring, model self-evaluation, and Cliff’s δ effect size analysis, they demonstrate that reducing constraint-related tokens by 71%—and overall prompt length by 25–30%—does not significantly degrade adherence performance. Crucially, compliance is found to depend primarily on constraint type and task domain rather than prompt format, underscoring the importance of deliberate structured constraint design.

This study systematically investigates the impact of programming language choice on token consumption during code generation by reasoning agents. Through controlled experiments, it evaluates token efficiency across five state-of-the-art models solving problems of equivalent difficulty in Python, Java, Rust, and OCaml. The work introduces a multidimensional trajectory analysis framework integrating trace re-execution, test-result vector abstraction, intermediate solution annotation, and natural language analysis. It presents the first quantitative assessment of cross-language token usage disparities among agents, revealing that language familiarity significantly influences generation behavior: in less familiar languages, agents are more prone to producing uncompilable code, redundantly modifying already correct solutions, and resorting to Python prototyping to circumvent direct implementation in the target language.

This work addresses the challenge that large language model–based code agents struggle to construct complete, consistent, and verifiable cross-component functional chains over extended development cycles, often leading to misalignment between design and implementation. To mitigate this, the paper introduces CodeSpec, a novel approach featuring a dual-executable specification mechanism: it generates functional chains through semantic-architecture pairing and compiles them into complementary architectural and behavioral specifications, thereby ensuring design completeness and implementation consistency. Integrating evidence-based functional chain construction with a collaborative large-model development framework, CodeSpec achieves pass rates of 70.7%, 55.0%, and 49.9% on FeatureBench using DeepSeek-V4-Pro, significantly outperforming baselines such as Claude Code, and demonstrates strong generalization on NL2Repo-Bench.

This work proposes an agent evaluation framework grounded in the principle that “compression is intelligence,” quantifying an agent’s intelligence by the number of bits required to reconstruct target objects under fixed task distributions, interfaces, and computational budgets. It systematically introduces an information-theoretic compression perspective into agent assessment for the first time, employing arithmetic coding, seed encoding, and fallback mechanisms to effectively measure how agent components—such as tool use, environmental interaction, and multi-step reasoning—contribute to reduced code length (i.e., decreased uncertainty). The approach is validated across diverse tasks including reversed text generation, chess move prediction, protein sequence modeling, retrieval-augmented question answering, and semantic story compression. Experimental results demonstrate that agents endowed with advanced capabilities consistently achieve significantly higher information efficiency.

It remains unclear whether current large language models genuinely understand program semantics in code generation, and there is a lack of systematic evaluation of their ability to generate executable behavioral specifications. This work proposes CodeSpecBench, the first benchmark supporting multi-granularity tasks at both function and repository levels, which expresses preconditions and postconditions as executable Python functions and emphasizes the correctness and completeness of specifications. Using an execution-driven protocol, it evaluates a model’s capacity to accept valid behaviors and reject invalid ones. Experiments across 15 state-of-the-art models reveal that the highest pass rate on repository-level tasks is only 20.2%, significantly lagging behind general code generation performance, thereby demonstrating that specification generation is substantially more challenging and that strong code generation capabilities do not equate to deep semantic understanding.