Constraint Tree Exploration for Learning from Language Feedback

2026-10-06

AI Summary

This study addresses the challenge in interactive learning where natural language feedback is prone to misinterpretation, often leading to the erroneous elimination of valid solutions. By modeling user intent as latent constraints, this work proposes the TRACE algorithm, which explores the feasible region via a tree structure and employs a feedback mechanism that distinguishes falsification from identification to verify and refine constraints, combined with a pure exploration strategy for efficient interpretation of linguistic feedback. Theoretically, the paper establishes high-probability coverage bounds that significantly reduce dependence on the candidate class size. Empirically, TRACE achieves an 86% success rate on the RecMovie benchmark, substantially outperforming the 48% baseline while demonstrating superior robustness.

Abstract

Natural-language feedback in interactive learning often explains why an action failed by pointing to violated requirements. Misinterpreting this feedback can lead an agent to rule out valid solutions. We study this setting by modeling user intent as latent constraints over an action space and formulating learning from language feedback as pure exploration over feasible regions. We introduce TRACE, an algorithm that organizes candidate constraints in a tree and tests each proposed refinement by generating actions that satisfy it. TRACE commits to the refinement only if the resulting feedback does not contradict it over repeated tests. We distinguish two ways of using the same feedback: (i) falsification, which detects contradictions to the constraint set currently being tested, and (ii) identification, which may additionally name a violated constraint. We prove high-probability coverage bounds with dependence on the candidate class size $H$ for TRACE-Falsification. With reliable identification, TRACE-Identification can replace this dependence by $K/p_{\mathrm{ext}}$, where $K$ is the number of latent constraints and $p_{\mathrm{ext}}$ lower-bounds the probability of extracting a missing true constraint from informative feedback. We evaluate TRACE across six language-feedback tasks. On RecMovie, TRACE-Identification achieves 73% and 86% final-output success under caps of 20 and 60 evaluated outputs, compared with at most 42% and 48% for the evaluated prompting baselines given the same feedback and output caps. Controlled identity-corruption experiments further show greater robustness than direct accumulation when the falsification detector remains reliable.

Problem

• language feedback
• constraint exploration
• interactive learning
• pure exploration
• user intent

Innovation

• Constraint Tree Exploration
• Language Feedback
• Pure Exploration
• Falsification and Identification
• Coverage Bounds