Iterative Policy Refinement through Semantic Rollout Analysis

2026-10-01

AI Summary

This study addresses the limitation of existing structured policy generation methods, which rely on manual or static knowledge and struggle to align with expert demonstrations. To overcome this, we propose a closed-loop iterative framework leveraging large language models (LLMs). The approach semanticizes rollout data into tabular formats, enabling LLMs to automatically diagnose and rectify structural deficiencies in policies. By utilizing tabularized rollout analysis as a feedback signal, the framework achieves automated alignment of policy structures without human intervention. Furthermore, this work integrates imitation learning with automated code generation techniques. Experimental results demonstrate that the proposed method improves performance by 15% while reducing computational costs by 75%, significantly enhancing overall policy generation efficiency.

Abstract

Structured policies improve efficiency, robustness, and interpretability in imitation learning by introducing task-specific inductive bias, but existing structure generation methods rely either on extensive human input or on static domain knowledge encoded in LLMs, which may be inconsistent with the expert demonstrations. We propose a closed-loop framework that iteratively refines structured policies using LLM-guided analysis of policy rollouts. By logging rollouts as semantically meaningful tabular data and prompting the LLM to generate diagnostic analysis code, our method identifies suboptimalities in the policy structure and iteratively corrects them without requiring human instruction. Experiments on car racing and door opening tasks show that our approach improves imitation learning performance by up to 15% over zero-shot LLM-generated structures and requires 75% less compute to achieve the same reinforcement learning performance. These results demonstrate that tabular rollout analysis provides an effective feedback signal to align LLM-generated policy structures with expert demonstrations, and we can utilize it to generate good policy structures automatically.

Problem

• Imitation Learning
• Structured Policies
• Large Language Models
• Policy Generation

Innovation

• Iterative Policy Refinement
• Semantic Rollout Analysis
• Closed-loop Framework
• Imitation Learning
• LLM-guided Diagnosis