Getting Better at Working With You: Compiling User Corrections into Runtime Enforcement for Coding Agents

2026-06-11

AI Summary

This work addresses the challenge that interactive large language model agents often fail to consistently adhere to user preferences across multi-turn dialogues, repeatedly committing the same errors. To overcome this limitation, the authors propose TRACE, a novel framework that automatically distills user correction statements into atomic formal rules and compiles them into runtime enforcement checks, thereby shifting from passive memory to active constraint-based adherence. The approach integrates rule mining, natural language-to-formal-rule rewriting, and seamless embedding within the agent’s execution pipeline. Experimental results demonstrate significant improvements: on the ClawArena benchmark, preference violation rates drop to 37.6% in-distribution and 2.0% out-of-distribution; on a MemoryArena-derived task, violations decrease to 60.5%, with task success rates surpassing those of the strongest memory-based baseline.

Abstract

Interactive LLM agents are becoming part of daily work, but they do not reliably become easier to work with over time: a correction remembered in one session may still be violated in the next. We study this gap between preference access and preference compliance. In tasks derived from anonymized real-user friction cases, Mem0 memory still leaves 57.5% of applicable preference checks violated. We introduce Test-time Rule Acquisition and Compiled Enforcement (TRACE), a drop-in skill-layer pipeline for coding-agent runtimes that mines user corrections, rewrites them as atomic rules, and compiles them into runtime checks that must pass before an agent completes future tasks. Unlike runtime checks written ahead of time by developers, TRACE skills come from the user's own chat corrections. We evaluate TRACE with simulated user-in-the-loop experiments on ClawArena coding-agent tasks and MemoryArena-derived memory-intensive tasks. On ClawArena, TRACE reduces held-out preference violation from 100.0% to 37.6% on in-distribution tasks and from 100.0% to 2.0% on out-of-distribution tasks. On MemoryArena-derived tasks, TRACE reduces in-distribution violation from 100.0% to 60.5% while matching or exceeding the strongest memory baseline on task pass. These results suggest that compiling corrections into runtime enforcement can address a repeated-friction failure mode that memory alone does not reliably solve, reducing the need for users to restate the same correction across future sessions. Experiment code is available at https://github.com/YujunZhou/TRACE_exp, and the deployable skill is available at https://github.com/YujunZhou/tellonce.

Problem

• preference compliance
• user corrections
• interactive LLM agents
• runtime enforcement
• repeated-friction

Innovation

• runtime enforcement
• user corrections
• coding agents
• preference compliance
• TRACE