The Invisible Lottery: How Subtle Cues Steer Algorithm Choice in LLM Code Generation

2026-06-02

AI Summary

This study addresses the susceptibility of large language models (LLMs) to subtle, task-irrelevant cues in prompts during code generation, which can inadvertently steer model outputs toward different algorithmic implementations—leading to uncontrolled variations in performance, security, and maintainability. The work introduces and empirically validates the phenomenon of “algorithmic steering,” demonstrating through 46,535 controlled trials across 11 programming tasks that 19 categories of prompt cues can shift algorithm selection by up to 100 percentage points. To mitigate this issue, the authors propose explicitly naming the target algorithm in the prompt as a robust intervention, validating its effectiveness across multiple models, tasks, and ablation studies. Crucially, they show that even when all generated code passes identical test suites, the prompt context systematically governs algorithmic choice.

Abstract

Large language models (LLMs) now generate substantial production code, often for tasks with multiple valid algorithmic solutions. Incidental prompt cues, meaning contextual words or metadata outside the task specification, can steer which algorithm the model selects, even when all outputs pass the same tests. Prompt sensitivity is well studied as a tool to improve output quality. Here, output policy means algorithm choice under fixed correctness. We define algorithm steering as cue-induced shifts in algorithm-family distributions and run 46,535 controlled experiments across 11 tasks, 19 cue types (18 channels plus a memoization semantic-vs-surface ablation that preserves meaning while changing typography and punctuation), and 15 model configurations. We find large, systematic shifts in algorithm-family distributions (up to 100 pp), largely consistent with cue semantics, including in applied tasks such as rate limiting. Direct algorithm naming is the most reliable mitigation we tested. Accidental context therefore creates an "invisible lottery" over performance, security, and maintainability.

Problem

• algorithm steering
• prompt cues
• LLM code generation
• invisible lottery
• algorithm choice

Innovation

• algorithm steering
• prompt cues
• large language models
• code generation
• invisible lottery