The Living Thing / Notebooks :

Dimensionality reduction

Wherein I teach myself, amongst other things, how a sparse PCA works, and decide where to file multidimensional scaling.

Related: “clustering”? Also with the notion of similarity as seen in kernel tricks. Inducing a differential metric. Matrix factorisations and random features, high-dimensional statistics and discuss random projections and their role in compressed sensing etc.



PCA and cousins

Kernel PCA, linear algebra and probabilistic formulations.

Linear algebra version of PCA puts us in the world of matrix factorisationa.

Non-linear versions

  • additive component analysis:

    Principal component analysis (PCA) is one of the most versatile tools for unsupervised learning with applications ranging from dimensionality reduction to exploratory data analysis and visualization. While much effort has been devoted to encouraging meaningful representations through regularization (e.g. non-negativity or sparsity), underlying linearity assumptions can limit their effectiveness. To address this issue, we propose Additive Component Analysis (ACA), a novel nonlinear extension of PCA. Inspired by multivariate nonparametric regression with additive models, ACA fits a smooth manifold to data by learning an explicit mapping from a low-dimensional latent space to the input space, which trivially enables applications like denoising. Furthermore, ACA can be used as a drop-in replacement in many algorithms that use linear component analysis methods as a subroutine via the local tangent space of the learned manifold. Unlike many other nonlinear dimensionality reduction techniques, ACA can be efficiently applied to large datasets since it does not require computing pairwise similarities or storing training data during testing. Multiple ACA layers can also be composed and learned jointly with essentially the same procedure for improved representational power, demonstrating the encouraging potential of nonparametric deep learning. We evaluate ACA on a variety of datasets, showing improved robustness, reconstruction performance, and interpretability.

Autoencoder and word2vec

The nonlinear PCA” interpretation, I just heard from Junbin Gao.

\begin{equation*} L(x, x') = \|x-x\|^2=\|x-\simga(U*sigma*W^Tx+b)) + b')\|^2 \end{equation*}

Locality Preserving projections

Try to preserve the nearness of points if they are connected on some (weight) graph.

\begin{equation*} \sum_{i,j}(y_i-y_j)^2 w_{i,j} \end{equation*}

So we seen an optimal projection vector.

(requirement for sparse similarity matrix?)

Multidimensional scaling


Topological data analysis

Start with the distances between points and try to find a lower dimensional manifold which preserves their distances. Local MDS? TDB.

Random projection

see random embeddings

Stochastic neighbour embedding

Probabilisitically preserving closeness.