MACRO: Markov Chain Routing of Transformer Layers

2026-08-06

AI Summary

Standard large language models employ fixed-layer execution, which hinders dynamic adaptation of computation paths to input, thereby limiting efficiency and performance. This work proposes MACRO, a novel framework that formulates dynamic routing of Transformer layers as a context-dependent Markov decision process. MACRO adaptively selects operations—such as layer skipping, repetition, or residual addition—based on layer index, computational budget, directional displacement, and operational context, without requiring model fine-tuning or test-time labels. By incorporating top-k Viterbi decoding, MACRO efficiently searches high-probability routing paths, substantially reducing computational overhead. Experiments demonstrate that MACRO improves average accuracy by 5.0% across multiple open-source large models—with even greater gains for smaller models—and outperforms the current state-of-the-art method, Dr. LLM, by 7.2% in accuracy while reducing routing search time by 9.4× (from 14.8 to 1.6 hours).

Abstract

Standard Large Language Models (LLMs) execute layers sequentially. Dynamic layer routing, i.e. search for a different execution path through layers involving layer repetitions, skips and other moves, can improve performance. Existing routing approaches often require updating model weights, running expensive search loops per test instance, or demand ground-truth labels during inference. In this work, we propose Markov Chain Routing of Transformer Layers (MACRO), a framework that learns task-specific routes over LLM architectures without modifying underlying parameters. MACRO models layer routing as a context-dependent Markov policy conditioned on layer indices, computation budget phases, directional displacements, and operator context, supporting skip, repeat, and residual hidden-state addition operations. The Markov route distribution is updated via feedback on training data and decoded using a top-k Viterbi algorithm to isolate high-probability candidate programs. We evaluate MACRO across diverse reasoning and knowledge benchmarks on multiple open-weight LLMs. MACRO achieves a +5.0% average accuracy improvement over the unrouted baselines, with largest gains on small models. We outperform the best dynamic routing approach Dr. LLM by +7.2%, while reducing route-search time 9.4x (from 14.8 to 1.6 hours). Our code is publicly available at https://github.com/Batorskq/MACRO.

Problem

• dynamic layer routing
• Large Language Models
• layer execution path
• parameter-free routing
• inference efficiency

Innovation

• dynamic layer routing
• Markov policy
• Transformer architecture
• parameter-free adaptation
• Viterbi decoding