Autonomous Learning From Success and Failure: Goal-Conditioned Supervised Learning with Negative Feedback

2025-09-03

AI Summary

In sparse-reward settings, reinforcement learning suffers from inefficient exploration, while goal-conditioned supervised learning (GCSL) enables self-imitation but is hindered by self-experience bias and a unidirectional learning mechanism that exclusively leverages successful trajectories. To address these limitations, we propose Contrastive GCSL (C-GCSL), the first framework to integrate contrastive learning into GCSL. C-GCSL constructs positive and negative sample pairs to enable agents to learn goal-directed policies from both successful and failed experiences, thereby mitigating bias and enabling bidirectional feedback utilization. Our method synergistically combines goal-conditioned supervision, self-imitation, and contrastive representation learning, significantly enhancing policy generalization and exploration robustness. Empirical evaluation on multiple sparse-reward benchmark tasks demonstrates that C-GCSL substantially outperforms GCSL and state-of-the-art RL baselines, achieving an average 23.6% improvement in task success rate—validating the efficacy and novelty of explicitly modeling failure experiences.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Search and Optimization: Learning to Search
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions

Abstract

Reinforcement learning faces significant challenges when applied to tasks characterized by sparse reward structures. Although imitation learning, within the domain of supervised learning, offers faster convergence, it relies heavily on human-generated demonstrations. Recently, Goal-Conditioned Supervised Learning (GCSL) has emerged as a potential solution by enabling self-imitation learning for autonomous systems. By strategically relabelling goals, agents can derive policy insights from their own experiences. Despite the successes of this framework, it presents two notable limitations: (1) Learning exclusively from self-generated experiences can exacerbate the agents' inherent biases; (2) The relabelling strategy allows agents to focus solely on successful outcomes, precluding them from learning from their mistakes. To address these issues, we propose a novel model that integrates contrastive learning principles into the GCSL framework to learn from both success and failure. Through empirical evaluations, we demonstrate that our algorithm overcomes limitations imposed by agents' initial biases and thereby enables more exploratory behavior. This facilitates the identification and adoption of effective policies, leading to superior performance across a variety of challenging environments.

Problem

• Addresses biases from self-imitation learning in autonomous agents
• Enables learning from both successful and failed experiences
• Improves exploratory behavior in sparse reward environments

Innovation

• Integrates contrastive learning into GCSL framework
• Learns from both success and failure outcomes
• Overcomes agent biases through exploratory behavior