Deterministic Policy Gradient for Learning Equilibrium in Time-Inconsistent Control Problems
时间不一致控制问题中学习均衡的确定性策略梯度
Xin Guo, Yijie Huang, Xiang Yu
AI总结 提出一种连续时间无模型强化学习算法,通过确定性策略梯度和内定点迭代学习时间不一致控制问题的均衡策略,并在均值-方差投资组合和非指数贴现跟踪投资组合中验证有效性。
详情
- Comments
- Keywords: Time-inconsistent control, two-stage reformulation, model-free continuous-time reinforcement learning, deterministic policy gradient, fixed point iteration
在本文中,我们开发了一种连续时间无模型强化学习算法,用于学习一般时间不一致控制问题中的确定性均衡策略。利用扩展的Hamilton-Jacobi-Bellman系统,我们将原始时间不一致问题转化为一个等价的两阶段问题。在第一阶段,对于给定的辅助函数,我们采用确定性策略梯度方法在辅助的时间一致控制问题中学习最优策略。在第二阶段,给定更新后的策略,我们利用内定点迭代和某些鞅特征来学习辅助函数。作为理论贡献,我们提供了一些温和的模型假设,并建立了内定点迭代的收敛性。通过在两阶段之间重复这种演员-评论家风格的迭代,我们的算法旨在以统一的方式学习不同时间不一致性来源下的均衡。该算法在两种经典的时间不一致金融应用中的优越有效性得到了说明:均值-方差投资组合管理和非指数贴现下的最优跟踪投资组合。
In this paper, we develop a continuous-time model-free reinforcement learning algorithm to learn deterministic equilibrium policies in general time-inconsistent control problems. Utilizing the extended Hamilton-Jacobi-Bellman system, we recast the original time-inconsistent problem into an equivalent two-stage problem. In the first stage, for given auxiliary functions, we employ the deterministic policy gradient approach to learn an optimal policy in an auxiliary time-consistent control problem. In the second stage, given the updated policy, we exploit the inner fixed point iterations and some martingale characterizations to learn the auxiliary functions. As a theoretical contribution, we provide some mild model assumptions and establish the convergence of inner fixed point iterations. By repeating this actor-critic style of iterations across two stages, our algorithm aims to learn the equilibrium under different sources of time-inconsistency in a unified manner. The superior effectiveness of the proposed algorithm are illustrated in two classical financial applications with time-inconsistency: mean-variance portfolio management and optimal tracking portfolio under non-exponential discounting.