The Living Thing / Notebooks :

Orthogonal basis decompositions

Fourier and friends

Transforming something (let us assume, discrete experimental data rather than abstract mathematical objects) into a new basis, as in the Fourier or (no longer orthogonal, Laplace or z-transforms), usually with a goal of handling some inconveniently curvy data as linear data plus a non-local curvy transformation. This is big in, e.g. signal processing and time series analysis.

Other names: Spectral methods, Hilbert space methods, basis expansions, orthogonal projection methods.


An orthogonal basis requires you to have a domain and inner product specified. If you choose a new inner product, you need a different base to be orthogonal in general. If you use the machinery of an implicit Hilbert space for a given inner product without troubling to define an explicit form for the “feature space”, you are doing the kernel trick.

Todo: discuss orthogonal measurements in terms of isometry properties.

General setting

Here I am especially interested in the discrete case, by which I mean that we are finding orthogonal bases for signal vectors of countable, often finite, length, e.g. if we are looking at spaces of 1 second of audio at 44kHz, then we consider spaces \(\mathbb{R}^{44100}\). If we are looking at RGB webcam images we might consider \(\mathbb{R}_+^{640\times 480 \times 3}\). Apparently this case can be related to the continuous case by considering general transforms on Locally Compact Abelian groups, via Pontryagin_duality.

See the following course notes

or the texts Rudi17, ReSt00.

Orthogonal basis zoo

For related ideas, see also

Miscellaneous stuff to remember


Bosq, D., & Blanke, D. (2007) Inference and prediction in large dimensions. . Chichester, England ; Hoboken, NJ: John Wiley/Dunod
Gabardo, J.-P. (2000) Hilbert spaces of distributions having an orthogonal basis of exponentials. Journal of Fourier Analysis and Applications, 6(3), 277–298. DOI.
Jones, L. K.(1992) A simple lemma on greedy approximation in Hilbert space and convergence rates for projection pursuit regression and neural network training. The Annals of Statistics, 20(1), 608–613.
Mailhé, B., Gribonval, R., Vandergheynst, P., & Bimbot, F. (2011) Fast orthogonal sparse approximation algorithms over local dictionaries. Signal Processing, 91(12), 2822–2835. DOI.
Reiter, H., & Stegeman, J. D.(2000) Classical harmonic analysis and locally compact groups. . Courier Corporation
Rudin, W. (2017) Fourier analysis on groups. . Courier Dover Publications
Unser, M. A., & Tafti, P. (2014) An introduction to sparse stochastic processes. . New York: Cambridge University Press
Wang, Y.-X., Smola, A., & Tibshirani, R. J.(2014) The Falling Factorial Basis and Its Statistical Applications. ArXiv:1405.0558 [Stat].
Wiatowski, T., & Bölcskei, H. (2015) A Mathematical Theory of Deep Convolutional Neural Networks for Feature Extraction. ArXiv:1512.06293 [Cs, Math, Stat].