Hessian-informed, Coordinate Friendly Hamiltonian Monte Carlo in Linear Time
Hessian感知的、坐标友好的线性时间哈密顿蒙特卡洛
Son Luu, Nikola Surjanovic, Zuheng Xu, Trevor Campbell, Alexandre Bouchard-Côté
AI总结 提出一种将Riemannian哈密顿蒙特卡洛(RHMC)固定点迭代的计算复杂度从O(d^2)降至O(d)的方法,适用于具有“坐标友好”结构的目标分布,包括广义线性模型及稠密/稀疏图模型。
详情
Riemannian哈密顿蒙特卡洛(RHMC)是一种有前景的MCMC方法,因为它能够适应位置相关的预处理和多步提议。虽然RHMC在低维表现良好,但在高维中由于每次固定点迭代需要$O(d^3)$的成本而变得不可行,其中$d$是目标密度的维数。即使位置相关的预处理器基于Hessian的对角线,每次固定点迭代的成本仍然是$O(d^2)$。在本文中,我们提出一种计算方法,对于具有“坐标友好”结构的目标,将对角预处理器的RHMC固定点迭代的计算复杂度从$O(d^2)$降低到$O(d)$。这类分布包括广义线性模型以及其他稠密和稀疏图模型。该方法表示为操作计算图,因此可以自动化地处理黑盒目标。最后,我们通过实验证明,与使用位置无关和位置相关预处理器的先进HMC NUTS算法相比,我们的RHMC实现在各种目标分布上每单位计算时间产生了更好的样本质量。
Riemannian Hamiltonian Monte Carlo (RHMC) is a promising MCMC methodology thanks to its ability to accommodate position-dependent preconditioning and multi-step proposals. While RHMC performs well in low dimensions, it becomes infeasible in high dimensions due to its $O(d^3)$ cost per fixed-point iteration, where $d$ is the dimension of the target density. Even when the position-dependent preconditioner is based on the diagonal of the Hessian, the cost is still $O(d^2)$ per fixed-point iteration. In this paper, we propose a computational method to reduce the computational complexity of RHMC fixed-point iterations with diagonal preconditioners from $O(d^2)$ to $O(d)$ for targets with ``coordinate friendly'' structures. This distribution class includes generalized linear models as well as other dense and sparse graphical models. The method is expressed as manipulating the compute graph and can therefore be automated to work on black box targets. Finally, we show empirically that our implementation of RHMC results in better sample quality per unit of compute time for various target distributions compared to state-of-the-art HMC NUTS algorithms with both position-independent and position-dependent preconditioners.