Verifiable Self-Evolution for Open-Ended Dialogue Skills via Future-Feedback Prediction

2026-07-21

AI Summary

This work addresses the challenge of unstable evaluation signals in open-domain dialogue self-evolution, where user feedback dynamically shifts in response to model replies. To overcome this, the authors propose a future feedback prediction mechanism that reframes the self-evolution objective from generating responses to predicting whether a given reply will elicit positive subsequent user feedback. This reformulation enables a verifiable offline learning task using fixed interaction logs. The approach integrates supervised learning, textual skill optimization, validation gating, feedback quality filtering, and data balancing strategies, allowing reproducible skill evolution without online deployment while retaining human evaluation for final validation. Evaluated on a proprietary sales assistant dataset, the method achieves over 75% accuracy in feedback prediction, and the evolved feedback-aware skills demonstrate interpretability, facilitating diagnosis and refinement of response strategies.

Abstract

Textual skills provide a lightweight way to improve frozen language-model agents, but their self-evolution normally requires a stable validation signal. Such signals are natural in mathematics or code, where an answer can be checked after it changes, yet are problematic in open-ended dialogue: changing the assistant response also changes the user's next reaction, so a logged reaction cannot directly evaluate a counterfactual response. We propose future-feedback skill evolution, which first redirects self-evolution from prescribing the current answer to predicting whether the observed answer will lead to a positive or negative subsequent user signal. This prediction task is verifiable on fixed logged tuples and therefore supports validation-gated textual optimization. The evolved feedback skill captures interpretable criteria for response quality and can subsequently serve as a diagnostic and optimization target for answer skills. On a proprietary, privacy-preserving sales-assistant dataset, careful quality filtering and a balanced resolved/unresolved split yield more than 75% prediction accuracy. Beyond this result, the central contribution is a formulation that converts otherwise moving conversational feedback into a fixed offline learning target, enabling reproducible skill evolution without placing every candidate skill in live traffic. We discuss the boundary between observational verification and counterfactual validity, and position the method as an offline optimization stage rather than a replacement for final human or online evaluation.

Problem

• open-ended dialogue
• self-evolution
• validation signal
• counterfactual evaluation
• conversational feedback

Innovation

• future-feedback prediction
• verifiable self-evolution
• offline skill optimization
• open-ended dialogue
• counterfactual feedback