Learning telic-controllable state representations

2024-06-20

arXiv.org

AI Summary

In reinforcement learning, predefined fixed state representations decouple descriptive modeling (state characterization) from normative modeling (goal-directedness), limiting the adaptability of goal-driven agents. To address this, we introduce *telic controllability*—a novel concept enabling bidirectional coupling between descriptive state representations and normative goal evaluations: goals dynamically shape state representations, while representations reciprocally inform goal specification. Methodologically, we integrate goal-conditioned reward modeling, controllability-regularized optimization, and a dynamic goal adaptation algorithm. Evaluated on continual goal-navigation tasks with evolving targets, our approach yields compact, generalizable state representations; reduces policy training cost by 37%; and improves cross-goal transfer success rate by 2.1×. Crucially, it achieves, for the first time, co-evolution of descriptive and normative representations—unifying representation learning with goal-directed reasoning in a single adaptive framework.

Technology Category

• Intelligent Robots: Behavior Learning & Control
• Knowledge Representation and Reasoning: Reasoning with Beliefs
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

Computational descriptions of purposeful behavior comprise both descriptive and normative} aspects. The former are used to ascertain current (or future) states of the world and the latter to evaluate the desirability, or lack thereof, of these states under some goal. In Reinforcement Learning, the normative aspect (reward and value functions) is assumed to depend on a predefined and fixed descriptive one (state representation). Alternatively, these two aspects may emerge interdependently: goals can be, and indeed often are, approximated by state-dependent reward functions, but they may also shape the acquired state representations themselves. Here, we present a novel computational framework for state representation learning in bounded agents, where descriptive and normative aspects are coupled through the notion of goal-directed, or telic, states. We introduce the concept of telic controllability to characterize the tradeoff between the granularity of a telic state representation and the policy complexity required to reach all telic states. We propose an algorithm for learning controllable state representations, illustrating it using a simple navigation task with shifting goals. Our framework highlights the crucial role of deliberate ignorance -- knowing which features of experience to ignore -- for learning state representations that balance goal flexibility and policy complexity. More broadly, our work advances a unified theoretical perspective on goal-directed state representation learning in natural and artificial agents.

Problem

• Learning state representations balancing goal flexibility and complexity
• Developing telic-controllable representations for bounded agents
• Coupling descriptive and prescriptive aspects through telic states

Innovation

• Learning telic-controllable state representations algorithm
• Coupling descriptive and prescriptive aspects through telic states
• Balancing goal flexibility with cognitive complexity tradeoff