The Living Thing / Notebooks :

Gaussian processes

Usefulness: 🔧 🔧
Novelty: 💡
Uncertainty: 🤪 🤪 🤪
Incompleteness: 🚧 🚧 🚧

“Gaussian Processes” are stochastic processes/fields with jointly Gaussian distributions of observations. The most familiar of these to many of us is the Gauss-Markov process, a.k.a. the Wiener process. These processes are convenient due to certain useful properties of the multivariate Gaussian distribution (e.g. being uniquely specified by first and second moments, nice behaviour under various linear operations, kernel tricks…). Especially neat applications include Gaussian process regression and spatial statistics.

Gaussian processes are, specifically, probabilistic distributions over random functions \(\mathcal{I}\to \mathbb{C}\) for some index (or argument) set \(\mathcal{I}\) often taken to be \(\mathcal{I}:=\mathbb{R}^d\).

We typically work with a centred (i.e. mean-zero) process, in which case for every finite set \(\mathbf{f}:=\{f(t_k);k=1,\dots,K\}\) of observations of that process, the joint distribution is centred Gaussian,

\[\begin{aligned} \mathbf{f}(t) &\sim \operatorname{GP}\left(0, \kappa(t, t';\mathbf{\theta})\right) \\ &\Rightarrow\\ p(\mathbf{f}) &=(2\pi )^{-{\frac {K}{2}}}\det({\boldsymbol {\mathrm{K} }})^{-{\frac {1}{2}}}\,e^{-{\frac {1}{2}}\mathbf {f}^{\!{\mathsf {T}}}{\boldsymbol {\mathrm{K} }}^{-1}\mathbf {f}}\\ &=\mathcal{N}(\mathbf{f};0, \mathrm{K}). \end{aligned}\] where \(\mathrm{K}\) is the sample covariance matrix defined such that its entries are given by \(\mathrm{K}_{jk}=\kappa(t_j,t_k).\) That is, this is the covariance kernel that maps from function argument\(t\) – to distributions over function values. In this case, we are specifying only the second moments and this is giving us all the remaining properties of the process.

Relationship between addition of covariance kernels and of processes

🚧

Refs

Abrahamsen, Petter. 1997. “A Review of Gaussian Random Fields and Correlation Functions.” http://publications.nr.no/publications.nr.no/directdownload/publications.nr.no/rask/old/917_Rapport.pdf.

Bochner, Salomon. 1959. Lectures on Fourier Integrals. Princeton University Press. http://books.google.com?id=MWCYDwAAQBAJ.

Khintchine, A. 1934. “Korrelationstheorie der stationären stochastischen Prozesse.” Mathematische Annalen 109 (1): 604–15. https://doi.org/10.1007/BF01449156.

Rasmussen, Carl Edward, and Christopher K. I. Williams. 2006. Gaussian Processes for Machine Learning. Adaptive Computation and Machine Learning. Cambridge, Mass: MIT Press. http://www.gaussianprocess.org/gpml/.

Yaglom, A. M. 1987. Correlation Theory of Stationary and Related Random Functions: Supplementary Notes and References. Springer Series in Statistics. New York, NY: Springer Science & Business Media.