The Living Thing / Notebooks :

Deep learning as a dynamical system

Image credit: Donny Darko
Image credit: Donny Darko

A recurring movement within deep learning research which tries to render the learning of prediction functions tractable by considering them as dynamical systems, and using the theory of stability in the context of Hamiltonians,r optimal control and/or ODE solvers, to make it all work.

I’ve been interested by this ever since seeing the Haber and Ruthotto paper, but it’s got a real kick recently since the Vector Institute team’s paper won the prize at NeurIPS for the ODE formulation of the problem.

Stability of training

Related, but not quite the same, notion of stability, as in data-stability in learning.

Can it work on time series?

Good question; It looks like it should, since there is an implicit time series the ODE-solver. But these problems so far have use non-time-series data.

Neural ODE regressors

By which I mean learning an ODE whose solution is the regression problem, which is a particular case of learning an ODE. There are various layperons’ introductions to this, including the simple and practical magical take in julia.

Random stuff

My question: How can this be made Bayesian?

Is this bit of overwrought? Improving Neural Models by Compensating for Discrete Rather Than Continuous Filter Dynamics when Simulating on Digital Systems.

TBC. Lyapunov analysis, Hamiltonian dynamics.