From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models

2026-07-29

AI Summary

Autoregressive language models struggle with non-causal discrete reasoning tasks such as code generation due to their inherent inability to perform arbitrary-order inference. This work identifies, for the first time, the “positional uncertainty” problem that arises between arbitrary-order interfaces and reasoning processes. To bridge this gap, the authors propose two complementary approaches: insertion-based masked diffusion (FlexMDM), which enables flexible token insertion at arbitrary positions, and latent-space masked diffusion, which supports semantic chunk-level prediction. Experimental results demonstrate significant performance gains on Python code generation using a 7B-parameter model and on GSM8K mathematical reasoning with a 125M-parameter model, while also exhibiting diverse arbitrary-order reasoning behaviors.

Abstract

Many discrete reasoning tasks, such as code generation, are inherently non-causal: programmers move between high-level structure and local details, a process we call any-order inference. For autoregressive language models, which lack a native any-order interface, non-causal abilities such as infilling and next-edit prediction require hand-designed mechanisms. Can we instead design models that natively support any-order inference? Masked diffusion models have recently emerged as compelling candidates, as their any-order training objective naturally offers an any-order prediction interface. This interface, however, does not automatically yield any-order inference. We demonstrate that this interface-inference gap stems from positional uncertainty: fixed-canvas, token-level models may know what semantic component should appear without knowing where to place it. In light of this, we propose two complementary approaches: (1) Insertion-based masked diffusion, building on FlexMDM (Kim et al, 2025), relaxes fixed-position commitments via insertions, enabling generation across non-contiguous regions. (2) Latent-space masked diffusion shifts prediction to coarser semantic segments, enabling search over latent generation orders. Empirically, we train a 7B FlexMDM for Python coding and a 125M LatentMDM for GSM8K and show that both approaches induce distinct any-order inference behaviors and improve downstream performance. We release our codebase at https://github.com/SeunggeunKimkr/genuine-any-order.

Problem

• any-order inference
• non-causal reasoning
• masked diffusion models
• positional uncertainty
• discrete reasoning tasks

Innovation

• any-order inference
• masked diffusion models
• positional uncertainty
• insertion-based generation
• latent-space diffusion