The Living Thing / Notebooks :

Natural Gradient descent

Climbing slower on the tricky bits

Usefulness: 🔧
Novelty: 💡
Uncertainty: 🤪 🤪 🤪
Incompleteness: 🚧 🚧 🚧

A placeholder.

Gradient descent with the natural gradient, or a close approximation thereto. Presumably related: Information geometry. It seems to related to 2nd order optimisation, although I woudl need to read past the abstract to discover how.

What is that natural gradient then? Should work it out.

Martens says

Natural gradient descent is an optimization method traditionally motivated from the perspective of information geometry, and works well for many applications as an alternative to stochastic gradient descent. In this paper we critically analyze this method and its properties, and show how it can be viewed as a type of approximate 2nd-order optimization method, where the Fisher information matrix can be viewed as an approximation of the Hessian. This perspective turns out to have significant implications for how to design a practical and robust version of the method. Additionally, we make the following contributions to the understanding of natural gradient and 2nd-order methods: a thorough analysis of the convergence speed of stochastic natural gradient descent (and more general stochastic 2nd-order methods) as applied to convex quadratics, a critical examination of the oft-used “empirical” approximation of the Fisher matrix, and an analysis of the (approximate) parameterization invariance property possessed by natural gradient methods, which we show still holds for certain choices of the curvature matrix other than the Fisher, but notably not the Hessian.

Other people’s study blogs:

Natural Policy Gradient

What the reinforcement learning people do? A brutally short explanation here, and a longer informal one here, and a downright lengthy one here.

Refs

Amari, Shun-ichi. 1998. “Natural Gradient Works Efficiently in Learning.” Neural Computation 10 (2): 251–76. https://doi.org/10.1162/089976698300017746.

Amari, Shun-ichi, Ryo Karakida, and Masafumi Oizumi. 2018. “Fisher Information and Natural Gradient Learning of Random Deep Networks,” August. http://arxiv.org/abs/1808.07172.

Amari, Shun-ichi, Hyeyoung Park, and Kenji Fukumizu. 2000. “Adaptive Method of Realizing Natural Gradient Learning for Multilayer Perceptrons.” Neural Computation 12 (6): 1399–1409. https://doi.org/10.1162/089976600300015420.

Kakade, Sham M. 2002. “A Natural Policy Gradient.” In Advances in Neural Information Processing Systems, 8.

Martens, James. 2014. “New Insights and Perspectives on the Natural Gradient Method,” December. http://arxiv.org/abs/1412.1193.

Ollivier, Yann. 2017. “Online Natural Gradient as a Kalman Filter,” March. http://arxiv.org/abs/1703.00209.

Salimbeni, Hugh, Stefanos Eleftheriadis, and James Hensman. 2018. “Natural Gradients in Practice: Non-Conjugate Variational Inference in Gaussian Process Models,” March. http://arxiv.org/abs/1803.09151.

Schraudolph, Nicol N. 2002. “Fast Curvature Matrix-Vector Products for Second-Order Gradient Descent.” Neural Computation 14 (7): 1723–38. https://doi.org/10.1162/08997660260028683.

Zhang, Guodong, Shengyang Sun, David Duvenaud, and Roger Grosse. 2017. “Noisy Natural Gradient as Variational Inference,” December. http://arxiv.org/abs/1712.02390.