Training Small LLMs as Spatial Multi-Agent Policies

2026-08-02

AI Summary

This work addresses the challenge that small frozen large language models (LLMs) struggle to achieve meaningful rewards in spatial cooperative games due to their limitation to low-level action outputs, hindering effective collaboration. To overcome this, the authors propose a multi-agent framework grounded in the options framework and macro-action Dec-POMDPs: each agent is equipped with a symbolic option library generated via code synthesis, along with automatically derived feasibility constraints to prune invalid options. A LoRA-finetuned small LLM serves as the option-level policy, trained using the PA-MAGROPO algorithm. Without any manual design or reward shaping, the method elevates four distinct frozen LLMs from zero reward to proficient performance across three cooperative games. Behavioral audits further reveal that increased rewards do not necessarily indicate genuine collaborative competence, underscoring the critical need for behavior-based evaluation.

Abstract

Training LLM-based multi-agent systems with multi-agent reinforcement learning is rapidly gaining traction, and a parallel line of work argues that such systems should be judged by their behavior, not only their reward. We take up both threads in spatial cooperative games, where small frozen LLMs prompted with low-level actions fail outright, earning zero reward. Guided by the options/semi-MDP framework---and, because option execution is asynchronous across agents, its multi-agent extension in macro-action Dec-POMDPs---we equip each game with a library of symbolic \emph{options}: typed, state-feasible, short-horizon behaviors executed by a symbolic planner. Each library is drafted by a frontier coding model from the game's source code; the feasibility guards that filter each menu are then synthesized mechanically from cheap random-policy burn-in rollouts---a guard is adopted only if it explains repeated execution failures while hiding no logged success---so no guard is authored, selected, or reward-tuned by hand. Each agent's LLM acts as its policy over options, with a private per-agent LoRA adapter trained by a per-agent variant of multi-agent GRPO (PA-MAGRPO); this lifts frozen bases from zero reward to competent play across three games and four small backbones. Behavioral audits then reveal that reward and cooperation decouple: a rising reward curve may simply mean that one agent has learned to run the entire task alone while its partner idles---cooperation emerges only when the task makes it necessary. Reward alone is thus an unreliable readout of cooperation; behavioral evaluation must sit alongside it.

Problem

• multi-agent reinforcement learning
• spatial cooperative games
• large language models
• cooperation evaluation
• reward misalignment

Innovation

• options framework
• symbolic macro-actions
• automatic guard synthesis
• LoRA-adapted LLM policies
• behavioral evaluation