Summary statistics which donâ€™t require you to keep all the data but which allow you to do inference nearly as well. e.g sufficient statistics in exponential families allow you to do do certain kind of inference perfectly without anything except summaries. Methods such as variational Bayes summarize data by maintaining a posterior density (usually a mixture models) as a summary of all the data, at some cost in accuracy.. I think of these as nearly sufficient statistics but there are other framings, as *data summarization* which I am going to note here for later reference.

- Approximate Bayesian Computation
*inducing sets*, as seen in Gaussian processes*coresets*as seen in Bayesian linear models- probabilistic deep learning possibly does this
- Bounded Memory Learning considers this from a computation complexity standpoint - which hypotheses can be learned from data subsets

TBC.

## Coresets

Bayesian. Solve an optimisation problem to minimise distance between posterior with all data and with a weighted subset.

https://github.com/trevorcampbell/bayesian-coresets

## Directly approximate log likelihood

See nearly sufficient statistics.

# Refs

Agrawal, Raj, Caroline Uhler, and Tamara Broderick. 2018. â€śMinimal I-MAP MCMC for Scalable Structure Discovery in Causal DAG Models.â€ť In *International Conference on Machine Learning*, 89â€“98. http://proceedings.mlr.press/v80/agrawal18a.html.

Bachem, Olivier, Mario Lucic, and Andreas Krause. 2017. â€śPractical Coreset Constructions for Machine Learning.â€ť *arXiv Preprint arXiv:1703.06476*. https://arxiv.org/abs/1703.06476.

â€”â€”â€”. 2015. â€śCoresets for Nonparametric Estimation - the Case of DP-Means.â€ť In *International Conference on Machine Learning*, 209â€“17. http://proceedings.mlr.press/v37/bachem15.html.

Broderick, Tamara, Nicholas Boyd, Andre Wibisono, Ashia C Wilson, and Michael I Jordan. 2013. â€śStreaming Variational Bayes.â€ť In *Advances in Neural Information Processing Systems 26*, edited by C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, 1727â€“35. Curran Associates, Inc. http://papers.nips.cc/paper/4980-streaming-variational-bayes.pdf.

Campbell, Trevor, and Tamara Broderick. 2017. â€śAutomated Scalable Bayesian Inference via Hilbert Coresets,â€ť October. http://arxiv.org/abs/1710.05053.

â€”â€”â€”. 2018. â€śBayesian Coreset Construction via Greedy Iterative Geodesic Ascent.â€ť In *International Conference on Machine Learning*, 698â€“706. http://proceedings.mlr.press/v80/campbell18a.html.

Cortes, Corinna, Vitaly Kuznetsov, Mehryar Mohri, and Scott Yang. 2016. â€śStructured Prediction Theory Based on Factor Graph Complexity.â€ť In *Advances in Neural Information Processing Systems 29*, edited by D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, 2514â€“22. Curran Associates, Inc. http://papers.nips.cc/paper/6485-structured-prediction-theory-based-on-factor-graph-complexity.pdf.

Hensman, James, Nicolo Fusi, and Neil D. Lawrence. 2013. â€śGaussian Processes for Big Data.â€ť In *Uncertainty in Artificial Intelligence*, 282. Citeseer.

Huggins, Jonathan, Ryan P Adams, and Tamara Broderick. 2017. â€śPASS-GLM: Polynomial Approximate Sufficient Statistics for Scalable Bayesian GLM Inference.â€ť In *Advances in Neural Information Processing Systems 30*, edited by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, 3611â€“21. Curran Associates, Inc. http://papers.nips.cc/paper/6952-pass-glm-polynomial-approximate-sufficient-statistics-for-scalable-bayesian-glm-inference.pdf.

Huggins, Jonathan H., Trevor Campbell, and Tamara Broderick. 2016. â€śCoresets for Scalable Bayesian Logistic Regression,â€ť May. http://arxiv.org/abs/1605.06423.

Huggins, Jonathan H., Trevor Campbell, MikoĹ‚aj Kasprzak, and Tamara Broderick. 2018a. â€śScalable Gaussian Process Inference with Finite-Data Mean and Variance Guarantees,â€ť June. http://arxiv.org/abs/1806.10234.

â€”â€”â€”. 2018b. â€śPractical Bounds on the Error of Bayesian Posterior Approximations: A Nonasymptotic Approach,â€ť September. http://arxiv.org/abs/1809.09505.

Titsias, Michalis K. 2009. â€śVariational Learning of Inducing Variables in Sparse Gaussian Processes.â€ť In *International Conference on Artificial Intelligence and Statistics*, 567â€“74. http://www.jmlr.org/proceedings/papers/v5/titsias09a/titsias09a.pdf.