Permutation-Robust Decision Modeling with Candidate-Independent Block-Causal Attention

2026-10-01

AI Summary

This study addresses the permutation sensitivity problem in decision-making models, where candidate action scores are influenced by their serialization order. To mitigate this issue, we propose a Candidate-Independent Block Causal Attention mechanism. Built upon the Gemma 3 and Qwen3 series of large language models, this method achieves permutation-invariant decision modeling by blocking inter-candidate information flow and resetting positional encodings. Experimental results demonstrate that the proposed architecture significantly reduces permutation sensitivity while maintaining competitive decision quality. These findings validate that candidate isolation serves as a core mechanism for achieving decision symmetry.

Abstract

Decision models often score a variable-sized set of candidate actions encoded in a single sequence. This setting is increasingly relevant for System 1 components inside generative systems, where candidates may be proposed or ordered differently across runs. Standard causal cross-encoding is expressive, but it can make a candidate's score depend on serialization order rather than on the underlying decision problem. We introduce candidate-independent block-causal attention, which preserves causal computation within the shared context and each candidate while blocking cross-candidate information flow and resetting candidate positions. We compare this architecture with standard causal attention and complementary invariant baselines across Gemma 3 1B, Qwen3 1.7B, and Qwen3 4B backbones. Candidate-independent attention consistently reduces permutation sensitivity while retaining competitive decision quality; ablations indicate that candidate isolation is the primary source of the effect, with position resetting completing the intended symmetry. A larger Qwen3-4B study further examines the behavior of the proposed architecture with substantially more training data. Code is available at the \href{https://github.com/guyAmit/ci-decision-models}{\textcolor{blue}{project repository}}, and the \href{https://huggingface.co/Guy-Amit/qwen3-4b-ci-decision-4096-poc}{\textcolor{blue}{Qwen3-4B model artifact}} is available on Hugging Face.

Problem

• decision modeling
• permutation robustness
• causal attention
• candidate scoring
• generative systems

Innovation

• Block-Causal Attention
• Permutation Robustness
• Decision Modeling
• Candidate Isolation
• Position Resetting