The Living Thing / Notebooks :

Covariance estimation for Gaussian processes

Estimating the thing that is always given to you by oracles in statistics homework assignments. The meat of Gaussian process regression. Obviously not just of Guassian processes. Loosely speaking, if your stochastic distribution has only 2 free parameters and you can do mean and covariance estimation, you’ve done the whole thing.

To consider:

Estimating the covariance, precision, concentration matrices of things. Turns about to be a lot more involved than estimating means in various ways and at various times. Long story.


Wishart priors.

Parametric covariance models

For spatial statistics I am told I should look up Matérn covariance matrices for a parametric covariance field.

To read


Abrahamsen, P. (1997) A review of Gaussian random fields and correlation functions.
Aragam, B., Gu, J., & Zhou, Q. (2017) Learning Large-Scale Bayesian Networks with the sparsebn Package. ArXiv:1703.04025 [Cs, Stat].
Azizyan, M., Krishnamurthy, A., & Singh, A. (2015) Extreme Compressive Sampling for Covariance Estimation. ArXiv:1506.00898 [Cs, Math, Stat].
Baik, J., Arous, G. B., & Péché, S. (2005) Phase Transition of the Largest Eigenvalue for Nonnull Complex Sample Covariance Matrices. The Annals of Probability, 33(5), 1643–1697.
Banerjee, O., Ghaoui, L. E., & d’Aspremont, A. (2008) Model selection through sparse maximum likelihood estimation for multivariate gaussian or binary data. Journal of Machine Learning Research, 9(Mar), 485–516.
Barnard, J., McCulloch, R., & Meng, X.-L. (2000) Modeling Covariance Matrices in Terms of Standard Deviations and Correlations, with Application to Shrinkage. Statistica Sinica, 10(4), 1281–1311.
Ben Arous, G., & Péché, S. (2005) Universality of local eigenvalue statistics for some sample covariance matrices. Communications on Pure and Applied Mathematics, 58(10), 1316–1357. DOI.
Cai, T. T., Zhang, C.-H., & Zhou, H. H.(2010) Optimal rates of convergence for covariance matrix estimation. The Annals of Statistics, 38(4), 2118–2144. DOI.
Chan, G., & Wood, A. T. A.(1999) Simulation of stationary Gaussian vector fields. Statistics and Computing, 9(4), 265–268. DOI.
Daniels, M. J., & Pourahmadi, M. (2009) Modeling covariance matrices via partial autocorrelations. Journal of Multivariate Analysis, 100(10), 2352–2363. DOI.
DIETRICH, C., & NEWSAM, G. (n.d.) Fast and Exact Simulation of Stationary Gaussian Processes Through Circulant Embedding of the Covariance Matrix.
Efron, B. (2010) Correlated z-values and the accuracy of large-scale statistical estimates. Journal of the American Statistical Association, 105(491), 1042–1055. DOI.
Friedman, J., Hastie, T., & Tibshirani, R. (2008) Sparse inverse covariance estimation with the graphical lasso. Biostatistics, 9(3), 432–441. DOI.
Fuentes, M. (2006) Testing for separability of spatial–temporal covariance functions. Journal of Statistical Planning and Inference, 136(2), 447–466. DOI.
Gneiting, T., Kleiber, W., & Schlather, M. (2010) Matérn Cross-Covariance Functions for Multivariate Random Fields. Journal of the American Statistical Association, 105(491), 1167–1177. DOI.
Gray, R. M.(2006) Toeplitz and Circulant Matrices: A Review. Foundations and Trends® in Communications and Information Theory, 2(3), 155–239. DOI.
Guinness, J., & Fuentes, M. (2016) Circulant Embedding of Approximate Covariances for Inference From Gaussian Data on Large Lattices. Journal of Computational and Graphical Statistics, 26(1), 88–97. DOI.
Hackbusch, W. (2015) Hierarchical Matrices: Algorithms and Analysis. (1st ed.). Heidelberg New York Dordrecht London: Springer Publishing Company, Incorporated
Hansen, C. B.(2007) Generalized least squares inference in panel and multilevel models with serial correlation and fixed effects. Journal of Econometrics, 140(2), 670–694. DOI.
Heinrich, C., & Podolskij, M. (2014) On spectral distribution of high dimensional covariation matrices. ArXiv:1410.6764 [Math].
Hsieh, C.-J., Sustik, M. A., Dhillon, I. S., & Ravikumar, P. D.(2014) QUIC: quadratic approximation for sparse inverse covariance estimation. Journal of Machine Learning Research, 15(1), 2911–2947.
Huang, J. Z., Liu, N., Pourahmadi, M., & Liu, L. (2006) Covariance matrix selection and estimation via penalised normal likelihood. Biometrika, 93(1), 85–98. DOI.
James, W., & Stein, C. (1961) Estimation with quadratic loss. In Proceedings of the fourth Berkeley symposium on mathematical statistics and probability (Vol. 1, pp. 361–379).
Janková, J., & van de Geer, S. (2015) Honest confidence regions and optimality in high-dimensional precision matrix estimation. ArXiv:1507.02061 [Math, Stat].
Khoromskij, B. N., Litvinenko, A., & Matthies, H. G.(2009) Application of hierarchical matrices for computing the Karhunen–Loève expansion. Computing, 84(1–2), 49–67. DOI.
Khoshgnauz, E. (2012) Learning Markov Network Structure using Brownian Distance Covariance. ArXiv:1206.6361 [Cs, Stat].
Krumin, M., & Shoham, S. (2009) Generation of Spike Trains with Controlled Auto- and Cross-Correlation Functions. Neural Computation, 21(6), 1642–1664. DOI.
Lam, C., & Fan, J. (2009) Sparsistency and Rates of Convergence in Large Covariance Matrix Estimation. Annals of Statistics, 37(6B), 4254–4278. DOI.
Ledoit, O., & Wolf, M. (2004) A well-conditioned estimator for large-dimensional covariance matrices. Journal of Multivariate Analysis, 88(2), 365–411. DOI.
Loh, W.-L. (1991) Estimating covariance matrices II. Journal of Multivariate Analysis, 36(2), 163–174. DOI.
MacKay, D. J. C.(2002) Gaussian Processes. In Information Theory, Inference & Learning Algorithms (p. Chapter 45). Cambridge University Press
Mardia, K. V., & Marshall, R. J.(1984) Maximum likelihood estimation of models for residual covariance in spatial regression. Biometrika, 71(1), 135–146. DOI.
Meinshausen, N., & Bühlmann, P. (2006) High-dimensional graphs and variable selection with the lasso. The Annals of Statistics, 34(3), 1436–1462. DOI.
Minasny, B., & McBratney, A. B.(2005) The Matérn function as a general model for soil variograms. Geoderma, 128(3–4), 192–207. DOI.
Nowak, W., & Litvinenko, A. (2013) Kriging and Spatial Design Accelerated by Orders of Magnitude: Combining Low-Rank Covariance Approximations with FFT-Techniques. Mathematical Geosciences, 45(4), 411–435. DOI.
Pébay, P. (2008) Formulas for robust, one-pass parallel computation of covariances and arbitrary-order statistical moments. Sandia Report SAND2008-6212, Sandia National Laboratories.
Powell, C. E.(2014) Generating realisations of stationary gaussian random fields by circulant embedding. Matrix, 2(2), 1.
Ramdas, A., & Wehbe, L. (2014) Stein Shrinkage for Cross-Covariance Operators and Kernel Independence Testing. ArXiv:1406.1922 [Stat].
Rasmussen, C. E., & Williams, C. K. I.(2006) Gaussian processes for machine learning. . Cambridge, Mass: MIT Press
Ravikumar, P., Wainwright, M. J., Raskutti, G., & Yu, B. (2011) High-dimensional covariance estimation by minimizing ℓ1-penalized log-determinant divergence. Electronic Journal of Statistics, 5, 935–980. DOI.
Rosenblatt, M. (1984) Asymptotic Normality, Strong Mixing and Spectral Density Estimates. The Annals of Probability, 12(4), 1167–1180. DOI.
Sampson, P. D., & Guttorp, P. (1992) Nonparametric estimation of nonstationary spatial covariance structure. Journal of the American Statistical Association, 87(417), 108–119.
Schäfer, J., & Strimmer, K. (2005) A shrinkage approach to large-scale covariance matrix estimation and implications for functional genomics. Statistical Applications in Genetics and Molecular Biology, 4, Article32. DOI.
Shao, X., & Wu, W. B.(2007) Asymptotic spectral theory for nonlinear time series. The Annals of Statistics, 35(4), 1773–1801. DOI.
Stein, M. L.(2005) Space-time covariance functions. Journal of the American Statistical Association, 100(469), 310–321. DOI.
Sun, Y., & Stein, M. L.(2016) Statistically and Computationally Efficient Estimating Equations for Large Spatial Datasets. Journal of Computational and Graphical Statistics, 25(1), 187–208. DOI.
Takemura, A. (1984) An Orthogonally Invariant Minimax Estimator of the Covariance Matrix of a Multivariate Normal Population. Tsukuba Journal of Mathematics, 8(2), 367–376.
Yuan, M., & Lin, Y. (2007) Model selection and estimation in the Gaussian graphical model. Biometrika, 94(1), 19–35. DOI.
Zhang, T., & Zou, H. (2014) Sparse precision matrix estimation via lasso penalized D-trace loss. Biometrika, 101(1), 103–120. DOI.