# Learning of manifolds

### Also topological data analysis; other hip names to follow

Usefulness: đź”§
Novelty: đź’ˇ
Uncertainty: đź¤Ş đź¤Ş đź¤Ş
Incompleteness: đźš§ đźš§ đźš§

As in â€“ handling your high-dimensional, or graphical, data by trying to discover a low(er)-dimensional manifold that contains it. That is, inferring a hidden constraint that happens to have the form of a smooth surface of some low-ish dimension. related: Learning on manifolds

There are a million different versions of this. Multidimensional scaling seems to be the oldest.

Tangential aside: in dynamical systems we talk about creating very high dimensional Takens embedding for state space reconstruction for arbitrary nonlinear dynamics. I imagine there are some connections between learning the lower-dimensional manifold upon which lies your data, and the higher dimensional manifold in which your dataâ€™s state space is naturally expressed. But I would not be the first person to notice this, so hopefully itâ€™s done for me somewhere?

See also kernel methods, which do regression on an implicit manifold, (how do you reconcile these, btw?) and functional regression where the manifold isnâ€™t even necessarily low dimensional, although typically still smooth, in some sense.

See also information geometry, which doesnâ€™t give you a manifold for your data, but a manifold in which the parametric model itself is embedded.

To look at: ISOMAP, Locally linear embedding, spectral embeddings, multidimensional scalingâ€¦

Bioinformatics is leading to some weird use of data manifolds; see for example BeDY16 for the performance implications of knowing the manifold shape for *-omics search, using compressive manifold storage based on both fractal dimension and metric entropy concepts. Also suggestive connection with fitness landscape in evolution.

Neural networks have some implicit manifolds, if you squint right. see Christopher Olahsâ€™s visual explanation how, whose diagrams should be stolen by someone trying to explain V-C dimension.

MoSF13 argue:

Manifold learning algorithms have recently played a crucial role in unsupervised learning tasks such as clustering and nonlinear dimensionality reduction[â€¦] Many such algorithms have been shown to be equivalent to Kernel PCA (KPCA) with data dependent kernels, itself equivalent to performing classical multidimensional scaling (cMDS) in a high dimensional feature space (SchĂ¶lkopf et al., 1998; Williams, 2002; Bengio et al., 2004).[â€¦] Recently, it has been observed that the majority of manifold learning algorithms can be expressed as a regularized loss minimization of a reconstruction matrix, followed by a singular value truncation (Neufeld et al., 2012)

## Implementations

### TTK

TTK

The Topology ToolKit (TTK) is an open-source library and software collection for topological data analysis in scientific visualization.

TTK can handle scalar data defined either on regular grids or triangulations, either in 2D or in 3D. It provides a substantial collection of generic, efficient and robust implementations of key algorithms in topological data analysis. It includes:

• For scalar data: critical points, integral lines, persistence diagrams, persistence curves, merge trees, contour trees, Morse-Smale complexes, topological simplification;

• For bivariate scalar data: fibers, fiber surfaces, continuous scatterplots, Jacobi sets, Reeb spaces;

• For uncertain scalar data: mandatory critical points;

### scikit-learn

scikit-learn implements a grab-bag of algorithms

### tapkee

C++: Tapkee. Pro-tip â€“ even without coding, tapkee does a long list of nice dimensionality reduction from the CLI, some of which are explicitly manifold learners (and the rest are matrix factorisations which is not so different)

• Locally Linear Embedding and Kernel Locally Linear Embedding (LLE/KLLE)
• Neighborhood Preserving Embedding (NPE)
• Local Tangent Space Alignment (LTSA)
• Linear Local Tangent Space Alignment (LLTSA)
• Hessian Locally Linear Embedding (HLLE)
• Laplacian eigenmaps
• Locality Preserving Projections
• Diffusion map
• Isomap and landmark Isomap
• Multidimensional scaling and landmark Multidimensional scaling (MDS/lMDS)
• Stochastic Proximity Embedding (SPE)
• PCA and randomized PCA
• Kernel PCA (kPCA)
• t-SNE
• Barnes-Hut-SNE

# Refs

Arjovsky, Martin, Soumith Chintala, and LĂ©on Bottou. 2017. â€śWasserstein Generative Adversarial Networks.â€ť In International Conference on Machine Learning, 214â€“23. http://proceedings.mlr.press/v70/arjovsky17a.html.

Aste, Tomaso, Ruggero Gramatica, and T Di Matteo. 2012. â€śExploring Complex Networks via Topological Embedding on Surfaces.â€ť Physical Review E 86 (3): 036109. https://doi.org/10.1103/PhysRevE.86.036109.

Aswani, Anil, Peter Bickel, and Claire Tomlin. 2011. â€śRegression on Manifolds: Estimation of the Exterior Derivative.â€ť The Annals of Statistics 39 (1): 48â€“81. https://doi.org/10.1214/10-AOS823.

Belkin, Mikhail, and Partha Niyogi. 2003. â€śLaplacian Eigenmaps for Dimensionality Reduction and Data Representation.â€ť Neural Computation 15 (6): 1373â€“96. https://doi.org/10.1162/089976603321780317.

Bengio, Yoshua, Aaron Courville, and Pascal Vincent. 2013. â€śRepresentation Learning: A Review and New Perspectives.â€ť IEEE Transactions on Pattern Analysis and Machine Intelligence 35: 1798â€“1828. https://doi.org/10.1109/TPAMI.2013.50.

Berger, Bonnie, Noah M. Daniels, and Y. William Yu. 2016. â€śComputational Biology in the 21st Century: Scaling with Compressive Algorithms.â€ť Communications of the ACM 59 (8): 72â€“80. https://doi.org/10.1145/2957324.

Carlsson, Gunnar, Tigran Ishkhanov, Vin de Silva, and Afra Zomorodian. 2008. â€śOn the Local Behavior of Spaces of Natural Images.â€ť International Journal of Computer Vision 76 (1): 1â€“12. https://doi.org/10.1007/s11263-007-0056-x.

Chen, Minhua, J. Silva, J. Paisley, Chunping Wang, D. Dunson, and L. Carin. 2010. â€śCompressive Sensing on Manifolds Using a Nonparametric Mixture of Factor Analyzers: Algorithm and Performance Bounds.â€ť IEEE Transactions on Signal Processing 58 (12): 6140â€“55. https://doi.org/10.1109/TSP.2010.2070796.

DeVore, Ronald A. 1998. â€śNonlinear Approximation.â€ť Acta Numerica 7 (January): 51â€“150. https://doi.org/10.1017/S0962492900002816.

Diaconis, Persi, and David Freedman. 1984. â€śAsymptotics of Graphical Projection Pursuit.â€ť The Annals of Statistics 12 (3): 793â€“815. http://www.jstor.org/stable/2240961.

â€”â€”â€”. 1986. â€śOn the Consistency of Bayes Estimates.â€ť The Annals of Statistics 14 (1): 1â€“26. http://www.jstor.org/stable/2241255.

Donoho, David L., and Carrie Grimes. 2003. â€śHessian Eigenmaps: Locally Linear Embedding Techniques for High-Dimensional Data.â€ť Proceedings of the National Academy of Sciences 100 (10): 5591â€“6. https://doi.org/10.1073/pnas.1031596100.

Freund, Yoav, Sanjoy Dasgupta, Mayank Kabra, and Nakul Verma. 2007. â€śLearning the Structure of Manifolds Using Random Projections.â€ť In Advances in Neural Information Processing Systems, 473â€“80. http://machinelearning.wustl.edu/mlpapers/paper_files/NIPS2007_133.pdf.

Gashler, Mike, and Tony Martinez. 2012. â€śRobust Manifold Learning with CycleCut.â€ť Connection Science 24 (1): 57â€“69. https://doi.org/10.1080/09540091.2012.664122.

Hadsell, R., S. Chopra, and Y. LeCun. 2006. â€śDimensionality Reduction by Learning an Invariant Mapping.â€ť In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2:1735â€“42. https://doi.org/10.1109/CVPR.2006.100.

Hall, Peter, and Ker-Chau Li. 1993. â€śOn Almost Linearity of Low Dimensional Projections from High Dimensional Data.â€ť The Annals of Statistics 21 (2): 867â€“89. http://www.jstor.org/stable/2242265.

Hawe, S., M. Kleinsteuber, and K. Diepold. 2013. â€śAnalysis Operator Learning and Its Application to Image Reconstruction.â€ť IEEE Transactions on Image Processing 22 (6): 2138â€“50. https://doi.org/10.1109/TIP.2013.2246175.

He, Xiaofei, and Partha Niyogi. 2003. â€śLocality Preserving Projections.â€ť In Proceedings of the 16th International Conference on Neural Information Processing Systems, 16:153â€“60. NIPSâ€™03. Cambridge, MA, USA: MIT Press. https://papers.nips.cc/paper/2359-locality-preserving-projections.pdf.

Huckemann, Stephan F., Peter T. Kim, Ja-Yong Koo, and Axel Munk. 2010. â€śMĂ¶bius Deconvolution on the Hyperbolic Plane with Application to Impedance Density Estimation.â€ť The Annals of Statistics 38 (4): 2465â€“98. https://doi.org/10.1214/09-AOS783.

Kemp, Charles, and Joshua B Tenenbaum. 2008. â€śThe Discovery of Structural Form.â€ť Proceedings of the National Academy of Sciences 105 (31): 10687â€“92. https://doi.org/10.1073/pnas.0802631105.

Lahiri, Subhaneil, Peiran Gao, and Surya Ganguli. 2016. â€śRandom Projections of Random Manifolds,â€ť July. http://arxiv.org/abs/1607.04331.

Maaten, Laurens van der, and Geoffrey Hinton. 2008. â€śVisualizing Data Using T-SNE.â€ť Journal of Machine Learning Research 9 (Nov): 2579â€“2605. http://www.jmlr.org/papers/v9/vandermaaten08a.html.

Moustafa, Karim Abou-, Dale Schuurmans, and Frank Ferrie. 2013. â€śLearning a Metric Space for Neighbourhood Topology Estimation: Application to Manifold Learning.â€ť In Journal of Machine Learning Research, 341â€“56. http://jmlr.org/proceedings/papers/v29/Moustafa13.html.

Mukherjee, Sayan, Qiang Wu, and Ding-Xuan Zhou. 2010. â€śLearning Gradients on Manifolds.â€ť Bernoulli 16 (1): 181â€“207. https://doi.org/10.3150/09-BEJ206.

Roweis, Sam T., and Lawrence K. Saul. 2000. â€śNonlinear Dimensionality Reduction by Locally Linear Embedding.â€ť Science 290 (5500): 2323â€“6. https://doi.org/10.1126/science.290.5500.2323.

Saul, Lawrence K., and Sam T. Roweis. 2003. â€śThink Globally, Fit Locally: Unsupervised Learning of Low Dimensional Manifolds.â€ť The Journal of Machine Learning Research 4 (December): 119â€“55. https://doi.org/10.1162/153244304322972667.

SchĂ¶lkopf, Bernhard, Alexander Smola, and Klaus-Robert MĂĽller. 1997. â€śKernel Principal Component Analysis.â€ť In Artificial Neural Networks â€” ICANNâ€™97, edited by Wulfram Gerstner, Alain Germond, Martin Hasler, and Jean-Daniel Nicoud, 583â€“88. Lecture Notes in Computer Science. Springer Berlin Heidelberg. https://doi.org/10.1007/BFb0020217.

â€”â€”â€”. 1998. â€śNonlinear Component Analysis as a Kernel Eigenvalue Problem.â€ť Neural Computation 10 (5): 1299â€“1319. https://doi.org/10.1162/089976698300017467.

Shaw, Blake, and Tony Jebara. 2009. â€śStructure Preserving Embedding.â€ť In Proceedings of the 26th Annual International Conference on Machine Learning, 937â€“44. ICML â€™09. New York, NY, USA: ACM. https://doi.org/10.1145/1553374.1553494.

Shieh, Albert D., Tatsunori B. Hashimoto, and Edoardo M. Airoldi. 2011. â€śTree Preserving Embedding.â€ť Proceedings of the National Academy of Sciences 108 (41): 16916â€“21. https://doi.org/10.1073/pnas.1018393108.

Smola, Alex J., Robert C. Williamson, Sebastian Mika, and Bernhard SchĂ¶lkopf. 1999. â€śRegularized Principal Manifolds.â€ť In Computational Learning Theory, edited by Paul Fischer and Hans Ulrich Simon, 214â€“29. Lecture Notes in Computer Science 1572. Springer Berlin Heidelberg. http://link.springer.com/chapter/10.1007/3-540-49097-3_17.

Song, Dongjin, and Dacheng Tao. 2010. â€śBiologically Inspired Feature Manifold for Scene Classification.â€ť IEEE Transactions on Image Processing: A Publication of the IEEE Signal Processing Society 19 (1): 174â€“84. https://doi.org/10.1109/TIP.2009.2032939.

Steinke, Florian, and Matthias Hein. 2009. â€śNon-Parametric Regression Between Manifolds.â€ť In Advances in Neural Information Processing Systems 21, 1561â€“8. Curran Associates, Inc. http://machinelearning.wustl.edu/mlpapers/paper_files/NIPS2008_0692.pdf.

Tenenbaum, Joshua B, Vin de Silva, and John C Langford. 2000. â€śA Global Geometric Framework for Nonlinear Dimensionality Reduction.â€ť Science, 2000. http://web.mit.edu/cocosci/Papers/sci_reprint.pdf.

Wang, Boyue, Yongli Hu, Junbin Gao, Yanfeng Sun, Haoran Chen, and Baocai Yin. 2017. â€śLocality Preserving Projections for Grassmann Manifold.â€ť In PRoceedings of IJCAI, 2017. http://arxiv.org/abs/1704.08458.

Weinberger, Kilian Q., Fei Sha, and Lawrence K. Saul. 2004. â€śLearning a Kernel Matrix for Nonlinear Dimensionality Reduction.â€ť In Proceedings of the Twenty-First International Conference on Machine Learning, 106. ICML â€™04. New York, NY, USA: ACM. https://doi.org/10.1145/1015330.1015345.

Williams, Christopher K. I. 2001. â€śOn a Connection Between Kernel PCA and Metric Multidimensional Scaling.â€ť In Advances in Neural Information Processing Systems 13, edited by T. K. Leen, T. G. Dietterich, and V. Tresp, 46:675â€“81. MIT Press. https://doi.org/10.1023/A:1012485807823.

Wu, Qiang, Justin Guinney, Mauro Maggioni, and Sayan Mukherjee. 2010. â€śLearning Gradients: Predictive Models That Infer Geometry and Statistical Dependence.â€ť The Journal of Machine Learning Research 11: 2175â€“98. http://dl.acm.org/citation.cfm?id=1859926.

Yin, M., J. Gao, and Z. Lin. 2016. â€śLaplacian Regularized Low-Rank Representation and Its Applications.â€ť IEEE Transactions on Pattern Analysis and Machine Intelligence 38 (3): 504â€“17. https://doi.org/10.1109/TPAMI.2015.2462360.

Yu, Yaoliang, James Neufeld, Ryan Kiros, Xinhua Zhang, and Dale Schuurmans. 2012. â€śRegularizers Versus Losses for Nonlinear Dimensionality Reduction: A Factored View with New Convex Relaxations.â€ť In ICML 2012. http://arxiv.org/abs/1206.6455.

Zhou, Tianyi, Dacheng Tao, and Xindong Wu. 2011. â€śManifold Elastic Net: A Unified Framework for Sparse Dimension Reduction.â€ť Data Mining and Knowledge Discovery 22 (3): 340â€“71. http://link.springer.com/article/10.1007/s10618-010-0182-x.

Zhu, Jun-Yan, Philipp KrĂ¤henbĂĽhl, Eli Shechtman, and Alexei A. Efros. 2016. â€śGenerative Visual Manipulation on the Natural Image Manifold.â€ť In Proceedings of European Conference on Computer Vision. http://arxiv.org/abs/1609.03552.