On Policy Stochasticity in Mutual Information Optimal Control of Linear Systems

2025-07-29

AI Summary

This paper addresses the mutual information optimal control problem (MIOCP) for discrete-time linear systems, establishing—for the first time—the theoretical relationship between the temperature parameter (regularization coefficient) and policy stochasticity. Unlike existing maximum-entropy frameworks, we employ rigorous optimal control theory to prove the existence of an optimal policy and derive a sufficient condition for the transition from stochastic to deterministic policies. An alternating optimization algorithm is proposed and validated via numerical experiments, confirming the monotonic influence of the temperature parameter on policy entropy. Results show that high temperatures promote exploratory, high-entropy policies, while low temperatures yield near-deterministic control; a critical temperature threshold explicitly characterizes this phase transition. This work fills a fundamental theoretical gap in MIOCP regarding the temperature–stochasticity linkage and provides a principled foundation for designing controllers with tunable robustness and exploration capability.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Mixed Discrete/Continuous Search
• Machine Learning: Optimization

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems

Abstract

In recent years, mutual information optimal control has been proposed as an extension of maximum entropy optimal control. Both approaches introduce regularization terms to render the policy stochastic, and it is important to theoretically clarify the relationship between the temperature parameter (i.e., the coefficient of the regularization term) and the stochasticity of the policy. Unlike in maximum entropy optimal control, this relationship remains unexplored in mutual information optimal control. In this paper, we investigate this relationship for a mutual information optimal control problem (MIOCP) of discrete-time linear systems. After extending the result of a previous study of the MIOCP, we establish the existence of an optimal policy of the MIOCP, and then derive the respective conditions on the temperature parameter under which the optimal policy becomes stochastic and deterministic. Furthermore, we also derive the respective conditions on the temperature parameter under which the policy obtained by an alternating optimization algorithm becomes stochastic and deterministic. The validity of the theoretical results is demonstrated through numerical experiments.

Problem

• Clarify relationship between temperature parameter and policy stochasticity
• Establish optimal policy existence for mutual information control
• Derive conditions for stochastic and deterministic optimal policies

Innovation

• Mutual information optimal control extension
• Temperature parameter conditions analysis
• Alternating optimization algorithm policy conditions