The Living Thing / Notebooks :

RKHS distribution embedding

The intersection of reproducing kernel methods, dependence tests <{filename}independence.rst>`_and `probability metrics; where you use a clever RKHS embedding to measure probability distributions.

A mere placeholder for now.

This abstract by Zoltán Szabó might serve to highlight some keywords.

Maximum mean discrepancy (MMD) and Hilbert-Schmidt independence criterion (HSIC) are among the most popular and successful approaches in applied mathematics to measure the difference and the independence of random variables, respectively. Thanks to their kernel-based foundations, MMD and HSIC are applicable on a large variety of domains such as documents, images, trees, graphs, time series, dynamical systems, sets or permutations. Despite their tremendous practical success, quite little is known about when HSIC characterizes independence and MMD with tensor kernel can discriminate probability distributions, in terms of the contributing kernel components. In this talk, I am going to provide a complete answer to this question, with conditions which are often easy to verify in practice. [Joint work with Bharath K. Sriperumbudur (PSU).

ITE toolbox (estimators)


Gretton, A., Fukumizu, K., Teo, C. H., Song, L., Schölkopf, B., & Smola, A. J.(2008) A Kernel Statistical Test of Independence. In Advances in Neural Information Processing Systems 20: Proceedings of the 2007 Conference. Cambridge, MA: MIT Press
Muandet et al. - 2016 - Kernel Mean Embedding of Distributions A Review a.pdf. (n.d.)
Muandet, K., Fukumizu, K., Sriperumbudur, B., Gretton, A., & Schölkopf, B. (2014) Kernel Mean Shrinkage Estimators. ArXiv:1405.5505 [Cs, Stat].
Muandet, K., Fukumizu, K., Sriperumbudur, B., & Schölkopf, B. (2017) Kernel Mean Embedding of Distributions: A Review and Beyond. Foundations and Trends® in Machine Learning, 10(1–2), 1–141. DOI.
Reid, M. D., & Williamson, R. C.(2009) Generalised Pinsker Inequalities. In arXiv:0906.1244 [cs, math].
Reid, M. D., & Williamson, R. C.(2011) Information, Divergence and Risk for Binary Experiments. Journal of Machine Learning Research, 12(Mar), 731–817.
Schölkopf, B., Muandet, K., Fukumizu, K., & Peters, J. (2015) Computing Functions of Random Variables via Reproducing Kernel Hilbert Space Representations. ArXiv:1501.06794 [Cs, Stat].
Sejdinovic, D., Sriperumbudur, B., Gretton, A., & Fukumizu, K. (2012) Equivalence of distance-based and RKHS-based statistics in hypothesis testing. The Annals of Statistics, 41(5), 2263–2291. DOI.
Smola, A., Gretton, A., Song, L., & Schölkopf, B. (2007) A Hilbert Space Embedding for Distributions. In M. Hutter, R. A. Servedio, & E. Takimoto (Eds.), Algorithmic Learning Theory (pp. 13–31). Springer Berlin Heidelberg
Song, L., Huang, J., Smola, A., & Fukumizu, K. (2009) Hilbert Space Embeddings of Conditional Distributions with Applications to Dynamical Systems. In Proceedings of the 26th Annual International Conference on Machine Learning (pp. 961–968). New York, NY, USA: ACM DOI.
Sriperumbudur, B. K., Fukumizu, K., Gretton, A., Schölkopf, B., & Lanckriet, G. R. G.(2012) On the empirical estimation of integral probability metrics. Electronic Journal of Statistics, 6, 1550–1599. DOI.
Sriperumbudur, B. K., Gretton, A., Fukumizu, K., Lanckriet, G., & Schölkopf, B. (2008) Injective Hilbert Space Embeddings of Probability Measures. In Proceedings of the 21st Annual Conference on Learning Theory (COLT 2008).
Sriperumbudur, B. K., Gretton, A., Fukumizu, K., Schölkopf, B., & Lanckriet, G. R. G.(2010) Hilbert Space Embeddings and Metrics on Probability Measures. Journal of Machine Learning Research, 11, 1517−1561.
Strobl, E. V., Zhang, K., & Visweswaran, S. (2017) Approximate Kernel-based Conditional Independence Tests for Fast Non-Parametric Causal Discovery. ArXiv:1702.03877 [Stat].
Szabo, Z., & Sriperumbudur, B. K.(2017) Characteristic and Universal Tensor Product Kernels. ArXiv:1708.08157 [Cs, Math, Stat].
Zhang, K., Peters, J., Janzing, D., & Schölkopf, B. (2012) Kernel-based Conditional Independence Test and Application in Causal Discovery. ArXiv:1202.3775 [Cs, Stat].
Zhang, Q., Filippi, S., Gretton, A., & Sejdinovic, D. (2016) Large-Scale Kernel Methods for Independence Testing. ArXiv:1606.07892 [Stat].