# Covariance estimation for stochastic processes

Usefulness: đź”§
Novelty: đź’ˇ
Uncertainty: đź¤Ş đź¤Ş đź¤Ş
Incompleteness: đźš§ đźš§ đźš§

Estimating the thing that is always given to you by oracles in statistics homework assignments, the covariance, precision, concentration matrices of things, or more generally, the covariance kernel. A complement to Gaussian process simulation. A thing that can make Gaussian process regression go better.

Estimating. Turns about to be a lot more involved than estimating means in various ways and at various times. Long story.

NB I am not doing a complete theory of covariance estimation here, just mentioning a couple of tidbits for future reference.

## Bayesian

Wishart priors đźš§

## Sandwich estimators

For robust covariances of vector data. AKA Heteroskedasticity-consistent covariance estimators. Incorporating Eicker-Huber-White sandwich estimator, Andrews kernel HAC estimator, Newey-West and others. For an intro see Achim Zeileis, Open-Source Econometric Computing in R.

# Refs

Aragam, Bryon, Jiaying Gu, and Qing Zhou. 2017. â€śLearning Large-Scale Bayesian Networks with the Sparsebn Package,â€ť March. http://arxiv.org/abs/1703.04025.

Azizyan, Martin, Akshay Krishnamurthy, and Aarti Singh. 2015. â€śExtreme Compressive Sampling for Covariance Estimation,â€ť June. http://arxiv.org/abs/1506.00898.

Baik, Jinho, GĂ©rard Ben Arous, and Sandrine PĂ©chĂ©. 2005. â€śPhase Transition of the Largest Eigenvalue for Nonnull Complex Sample Covariance Matrices.â€ť The Annals of Probability 33 (5): 1643â€“97.

Banerjee, Onureena, Laurent El Ghaoui, and Alexandre dâ€™Aspremont. 2008. â€śModel Selection Through Sparse Maximum Likelihood Estimation for Multivariate Gaussian or Binary Data.â€ť Journal of Machine Learning Research 9 (Mar): 485â€“516. http://www.jmlr.org/papers/v9/banerjee08a.html.

Barnard, John, Robert McCulloch, and Xiao-Li Meng. 2000. â€śModeling Covariance Matrices in Terms of Standard Deviations and Correlations, with Application to Shrinkage.â€ť Statistica Sinica 10 (4): 1281â€“1311. http://www3.stat.sinica.edu.tw/statistica/password.asp?vol=10&num=4&art=16.

Ben Arous, GĂ©rard, and Sandrine PĂ©chĂ©. 2005. â€śUniversality of Local Eigenvalue Statistics for Some Sample Covariance Matrices.â€ť Communications on Pure and Applied Mathematics 58 (10): 1316â€“57. https://doi.org/10.1002/cpa.20070.

Cai, T. Tony, Cun-Hui Zhang, and Harrison H. Zhou. 2010. â€śOptimal Rates of Convergence for Covariance Matrix Estimation.â€ť The Annals of Statistics 38 (4): 2118â€“44. https://doi.org/10.1214/09-AOS752.

Chan, G., and A. T. A. Wood. 1999. â€śSimulation of Stationary Gaussian Vector Fields.â€ť Statistics and Computing 9 (4): 265â€“68. https://doi.org/10.1023/A:1008903804954.

Chan, Tony F., Gene H. Golub, and Randall J. Leveque. 1983. â€śAlgorithms for Computing the Sample Variance: Analysis and Recommendations.â€ť The American Statistician 37 (3): 242â€“47. https://doi.org/10.1080/00031305.1983.10483115.

Cook, R. Dennis. 2018. â€śPrincipal Components, Sufficient Dimension Reduction, and Envelopes.â€ť Annual Review of Statistics and Its Application 5 (1): 533â€“59. https://doi.org/10.1146/annurev-statistics-031017-100257.

Cunningham, John P., Krishna V. Shenoy, and Maneesh Sahani. 2008. â€śFast Gaussian Process Methods for Point Process Intensity Estimation.â€ť In Proceedings of the 25th International Conference on Machine Learning, 192â€“99. ICML â€™08. New York, NY, USA: ACM Press. https://doi.org/10.1145/1390156.1390181.

Daniels, M. J., and M. Pourahmadi. 2009. â€śModeling Covariance Matrices via Partial Autocorrelations.â€ť Journal of Multivariate Analysis 100 (10): 2352â€“63. https://doi.org/10.1016/j.jmva.2009.04.015.

Dasgupta, Sanjoy, and Daniel Hsu. 2007. â€śOn-Line Estimation with the Multivariate Gaussian Distribution.â€ť In Learning Theory, edited by Nader H. Bshouty and Claudio Gentile, 4539:278â€“92. Berlin, Heidelberg: Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-540-72927-3_21.

Davies, Tilman M., and David Bryant. 2013. â€śOn Circulant Embedding for Gaussian Random Fields in R.â€ť Journal of Statistical Software 55 (9). https://doi.org/10.18637/jss.v055.i09.

Dietrich, C. R., and G. N. Newsam. 1993. â€śA Fast and Exact Method for Multidimensional Gaussian Stochastic Simulations.â€ť Water Resources Research 29 (8): 2861â€“9. https://doi.org/10.1029/93WR01070.

Efron, Bradley. 2010. â€śCorrelated Z-Values and the Accuracy of Large-Scale Statistical Estimates.â€ť Journal of the American Statistical Association 105 (491): 1042â€“55. https://doi.org/10.1198/jasa.2010.tm09129.

Friedman, Jerome, Trevor Hastie, and Robert Tibshirani. 2008. â€śSparse Inverse Covariance Estimation with the Graphical Lasso.â€ť Biostatistics 9 (3): 432â€“41. https://doi.org/10.1093/biostatistics/kxm045.

Fuentes, Montserrat. 2006. â€śTesting for Separability of Spatialâ€“Temporal Covariance Functions.â€ť Journal of Statistical Planning and Inference 136 (2): 447â€“66. https://doi.org/10.1016/j.jspi.2004.07.004.

Gneiting, Tilmann, William Kleiber, and Martin Schlather. 2010. â€śMatĂ©rn Cross-Covariance Functions for Multivariate Random Fields.â€ť Journal of the American Statistical Association 105 (491): 1167â€“77. https://doi.org/10.1198/jasa.2010.tm09420.

Gray, Robert M. 2006. â€śToeplitz and Circulant Matrices: A Review.â€ť Foundations and TrendsÂ® in Communications and Information Theory 2 (3): 155â€“239. https://doi.org/10.1561/0100000006.

Guinness, Joseph, and Montserrat Fuentes. 2016. â€śCirculant Embedding of Approximate Covariances for Inference from Gaussian Data on Large Lattices.â€ť Journal of Computational and Graphical Statistics 26 (1): 88â€“97. https://doi.org/10.1080/10618600.2016.1164534.

Hackbusch, Wolfgang. 2015. Hierarchical Matrices: Algorithms and Analysis. 1st ed. Springer Series in Computational Mathematics 49. Heidelberg New York Dordrecht London: Springer Publishing Company, Incorporated.

Hansen, Christian B. 2007. â€śGeneralized Least Squares Inference in Panel and Multilevel Models with Serial Correlation and Fixed Effects.â€ť Journal of Econometrics 140 (2): 670â€“94. https://doi.org/10.1016/j.jeconom.2006.07.011.

Heinrich, Claudio, and Mark Podolskij. 2014. â€śOn Spectral Distribution of High Dimensional Covariation Matrices,â€ť October. http://arxiv.org/abs/1410.6764.

Hsieh, Cho-Jui, MĂˇtyĂˇs A. Sustik, Inderjit S. Dhillon, and Pradeep D. Ravikumar. 2014. â€śQUIC: Quadratic Approximation for Sparse Inverse Covariance Estimation.â€ť Journal of Machine Learning Research 15 (1): 2911â€“47. http://www.jmlr.org/papers/volume15/hsieh14a/hsieh14a.pdf.

Huang, Jianhua Z., Naiping Liu, Mohsen Pourahmadi, and Linxu Liu. 2006. â€śCovariance Matrix Selection and Estimation via Penalised Normal Likelihood.â€ť Biometrika 93 (1): 85â€“98. https://doi.org/10.1093/biomet/93.1.85.

James, William, and Charles Stein. 1961. â€śEstimation with Quadratic Loss.â€ť In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, 1:361â€“79. http://projecteuclid.org/euclid.bsmsp/1200512173.

JankovĂˇ, Jana, and Sara van de Geer. 2015. â€śHonest Confidence Regions and Optimality in High-Dimensional Precision Matrix Estimation,â€ť July. http://arxiv.org/abs/1507.02061.

Kauermann, GĂ¶ran, and Raymond J. Carroll. 2001. â€śA Note on the Efficiency of Sandwich Covariance Matrix Estimation.â€ť Journal of the American Statistical Association 96 (456): 1387â€“96. https://www.jstor.org/stable/3085907.

Khoromskij, B. N., A. Litvinenko, and H. G. Matthies. 2009. â€śApplication of Hierarchical Matrices for Computing the Karhunenâ€“LoĂ¨ve Expansion.â€ť Computing 84 (1-2): 49â€“67. https://doi.org/10.1007/s00607-008-0018-3.

Khoshgnauz, Ehsan. 2012. â€śLearning Markov Network Structure Using Brownian Distance Covariance,â€ť June. http://arxiv.org/abs/1206.6361.

Krumin, Michael, and Shy Shoham. 2009. â€śGeneration of Spike Trains with Controlled Auto- and Cross-Correlation Functions.â€ť Neural Computation 21 (6): 1642â€“64. https://doi.org/10.1162/neco.2009.08-08-847.

Lam, Clifford, and Jianqing Fan. 2009. â€śSparsistency and Rates of Convergence in Large Covariance Matrix Estimation.â€ť Annals of Statistics 37 (6B): 4254â€“78. https://doi.org/10.1214/09-AOS720.

Ledoit, Olivier, and Michael Wolf. 2004. â€śA Well-Conditioned Estimator for Large-Dimensional Covariance Matrices.â€ť Journal of Multivariate Analysis 88 (2): 365â€“411. https://doi.org/10.1016/S0047-259X(03)00096-4.

Ling, Robert F. 1974. â€śComparison of Several Algorithms for Computing Sample Means and Variances.â€ť Journal of the American Statistical Association 69 (348): 859â€“66. https://doi.org/10.1080/01621459.1974.10480219.

Loh, Wei-Liem. 1991. â€śEstimating Covariance Matrices II.â€ť Journal of Multivariate Analysis 36 (2): 163â€“74. https://doi.org/10.1016/0047-259X(91)90055-7.

MacKay, David J C. 2002. â€śGaussian Processes.â€ť In Information Theory, Inference & Learning Algorithms, Chapter 45. Cambridge University Press. http://www.inference.phy.cam.ac.uk/mackay/itprnn/ps/534.548.pdf.

Mardia, K. V., and R. J. Marshall. 1984. â€śMaximum Likelihood Estimation of Models for Residual Covariance in Spatial Regression.â€ť Biometrika 71 (1): 135â€“46. https://doi.org/10.1093/biomet/71.1.135.

Meinshausen, Nicolai, and Peter BĂĽhlmann. 2006. â€śHigh-Dimensional Graphs and Variable Selection with the Lasso.â€ť The Annals of Statistics 34 (3): 1436â€“62. https://doi.org/10.1214/009053606000000281.

Minasny, Budiman, and Alex. B. McBratney. 2005. â€śThe MatĂ©rn Function as a General Model for Soil Variograms.â€ť Geoderma, Pedometrics 2003, 128 (3â€“4): 192â€“207. https://doi.org/10.1016/j.geoderma.2005.04.003.

Nowak, W., and A. Litvinenko. 2013. â€śKriging and Spatial Design Accelerated by Orders of Magnitude: Combining Low-Rank Covariance Approximations with FFT-Techniques.â€ť Mathematical Geosciences 45 (4): 411â€“35. https://doi.org/10.1007/s11004-013-9453-6.

PĂ©bay, Philippe. 2008. â€śFormulas for Robust, One-Pass Parallel Computation of Covariances and Arbitrary-Order Statistical Moments.â€ť Sandia Report SAND2008-6212, Sandia National Laboratories. http://prod.sandia.gov/techlib/access-control.cgi/2008/086212.pdf.

Pourahmadi, Mohsen. 2011. â€śCovariance Estimation: The GLM and Regularization Perspectives.â€ť Statistical Science 26 (3): 369â€“87. https://doi.org/10.1214/11-STS358.

Powell, Catherine E. 2014. â€śGenerating Realisations of Stationary Gaussian Random Fields by Circulant Embedding.â€ť Matrix 2 (2): 1.

Ramdas, Aaditya, and Leila Wehbe. 2014. â€śStein Shrinkage for Cross-Covariance Operators and Kernel Independence Testing,â€ť June. http://arxiv.org/abs/1406.1922.

Rasmussen, Carl Edward, and Christopher K. I. Williams. 2006. Gaussian Processes for Machine Learning. Adaptive Computation and Machine Learning. Cambridge, Mass: MIT Press. http://www.gaussianprocess.org/gpml/.

Ravikumar, Pradeep, Martin J. Wainwright, Garvesh Raskutti, and Bin Yu. 2011. â€śHigh-Dimensional Covariance Estimation by Minimizing â„“1-Penalized Log-Determinant Divergence.â€ť Electronic Journal of Statistics 5: 935â€“80. https://doi.org/10.1214/11-EJS631.

Rosenblatt, M. 1984. â€śAsymptotic Normality, Strong Mixing and Spectral Density Estimates.â€ť The Annals of Probability 12 (4): 1167â€“80. https://doi.org/10.1214/aop/1176993146.

Sampson, P D, and P Guttorp. 1992. â€śNonparametric Estimation of Nonstationary Spatial Covariance Structure.â€ť Journal of the American Statistical Association 87 (417): 108â€“19.

SchĂ¤fer, Juliane, and Korbinian Strimmer. 2005. â€śA Shrinkage Approach to Large-Scale Covariance Matrix Estimation and Implications for Functional Genomics.â€ť Statistical Applications in Genetics and Molecular Biology 4: Article32. https://doi.org/10.2202/1544-6115.1175.

Shao, Xiaofeng, and Wei Biao Wu. 2007. â€śAsymptotic Spectral Theory for Nonlinear Time Series.â€ť The Annals of Statistics 35 (4): 1773â€“1801. https://doi.org/10.1214/009053606000001479.

Stein, Michael L. 2005. â€śSpace-Time Covariance Functions.â€ť Journal of the American Statistical Association 100 (469): 310â€“21. https://doi.org/10.1198/016214504000000854.

Sun, Ying, and Michael L. Stein. 2016. â€śStatistically and Computationally Efficient Estimating Equations for Large Spatial Datasets.â€ť Journal of Computational and Graphical Statistics 25 (1): 187â€“208. https://doi.org/10.1080/10618600.2014.975230.

Takemura, Akimichi. 1984. â€śAn Orthogonally Invariant Minimax Estimator of the Covariance Matrix of a Multivariate Normal Population.â€ť Tsukuba Journal of Mathematics 8 (2): 367â€“76.

Whittle, P. 1952. â€śTests of Fit in Time Series.â€ť Biometrika 39 (3-4): 309â€“18. https://doi.org/10.1093/biomet/39.3-4.309.

â€”â€”â€”. 1953a. â€śThe Analysis of Multiple Stationary Time Series.â€ť Journal of the Royal Statistical Society. Series B (Methodological) 15 (1): 125â€“39.

â€”â€”â€”. 1953b. â€śEstimation and Information in Stationary Time Series.â€ť Arkiv FĂ¶r Matematik 2 (5): 423â€“34. https://doi.org/10.1007/BF02590998.

Whittle, Peter. 1952. â€śSome Results in Time Series Analysis.â€ť Scandinavian Actuarial Journal 1952 (1-2): 48â€“60. https://doi.org/10.1080/03461238.1952.10414182.

Yuan, Ming, and Yi Lin. 2007. â€śModel Selection and Estimation in the Gaussian Graphical Model.â€ť Biometrika 94 (1): 19â€“35. https://doi.org/10.1093/biomet/asm018.

Zeileis, Achim. 2004. â€śEconometric Computing with HC and HAC Covariance Matrix Estimators.â€ť Journal of Statistical Software 11 (10). https://doi.org/10.18637/jss.v011.i10.

â€”â€”â€”. 2006a. â€śImplementing a Class of Structural Change Tests: An Econometric Computing Approach.â€ť Computational Statistics & Data Analysis 50 (11): 2987â€“3008. https://doi.org/10.1016/j.csda.2005.07.001.

â€”â€”â€”. 2006b. â€śObject-Oriented Computation of Sandwich Estimators.â€ť Journal of Statistical Software 16 (1): 1â€“16. https://doi.org/10.18637/jss.v016.i09.

Zhang, T., and H. Zou. 2014. â€śSparse Precision Matrix Estimation via Lasso Penalized D-Trace Loss.â€ť Biometrika 101 (1): 103â€“20. https://doi.org/10.1093/biomet/ast059.