# Linear algebra

### If the thing is twice as big, the transformed version of the thing is also twice as big. {End}

Usefulness: đź”§
Novelty: đź’ˇ
Uncertainty: đź¤Ş đź¤Ş đź¤Ş
Incompleteness: đźš§ đźš§ đźš§

Oh! the hours I put in to studying the taxonomy and husbandry of matrices. Time has passed. I have forgotten much. Jacobians have begun to seem downright Old Testament.

And when you put the various operations of matrix calculus into the mix (derivative of trace of a skew-hermitian heffalump painted with a camel-hair brush) the combinatorial explosions of theorems and identities is intimidating.

Things I need:

## Basic linear algebra intros

• Kevin Brown on Bras, Kets, and Matrices

• Stanford CS229â€™s Linear Algebra Review and reference (PDF)

• fun, Tom Leinster, There are no non-trivial complex quarter turns but there are real ones, i.e.

for a linear operator T on a real inner product space,

$\langle T x, x \rangle = 0 \,\, \forall x \,\, \iff \,\, T^\ast = -T$

whereas for an operator on a complex inner product space,

$\langle T x, x \rangle = 0 \,\, \forall x \,\, \iff \,\, T = 0.$

Cool.

• Sheldon Axlerâ€™s Down with Determinants!. (Axle95) is a readable and intuitive introduction for undergrads:

Without using determinants, we will define the multiplicity of an eigenvalue and prove that the number of eigenvalues, counting multiplicities, equals the dimension of the underlying space. Without determinants, weâ€™ll define the characteristic and minimal polynomials and then prove that they behave as expected. Next, we will easily prove that every matrix is similar to a nice upper-triangular one. Turning to inner product spaces, and still without mentioning determinants, weâ€™ll have a simple proof of the finite-dimensional Spectral Theorem.

Determinants are needed in one place in the undergraduate mathematics curriculum: the change of variables formula for multi-variable integrals. Thus at the end of this paper weâ€™ll revive determinants, but not with any of the usual abstruse definitions. Weâ€™ll define the determinant of a matrix to be the product of its eigenvalues (counting multiplicities). This easy-to-remember definition leads to the usual formulas for computing determinants. Weâ€™ll derive the change of variables formula for multi-variable integrals in a fashion that makes the appearance of the determinant there seem natural.

He wrote a whole textbook on this basis, Axle14.

• a handy glossary is Mike Brooksâ€™ Matrix reference manual

• Singular Value Decomposition series, for its insight:

Most of the time when people talk about linear algebra even mathematicians), theyâ€™ll stick entirely to the linear map perspective or the data perspective, which is kind of frustrating when youâ€™re learning it for the first time. It seems like the data perspective is just a tidy convenience, that it justâ€śmakes senseâ€ť to put some data in a table. In my experience the singular value decomposition is the first time that the two perspectives collide, and (at least in my case) it comes with cognitive dissonance.

## Linear algebra and calculus

The multidimensional statistics/control theory workhorse.

See matrix calculus.

## Multilinear Algebra

Oooh you are playing with tensors? I donâ€™t have a bunch to say here but here is a compact explanation of Einstein summation, which turns out to be as simple as it needs to be, but no simpler.

# Refs

Alexander Graham. 1981. Kronecker Products and Matrix Calculus: With Applications. Horwood.

Axler, Sheldon. 1995. â€śDown with Determinants!â€ť The American Mathematical Monthly 102 (2): 139â€“54. https://doi.org/10.2307/2975348.

â€”â€”â€”. 2014. Linear Algebra Done Right. New York: Springer. http://dx.doi.org/10.1007/978-3-319-11080-6.

Boyd, Stephen P., and Lieven Vandenberghe. 2018. Introduction to Applied Linear Algebra: Vectors, Matrices, and Least Squares. Cambridge, UK ; New York, NY: Cambridge University Press.

Darrell A. Turkington. 2001. Matrix Calculus Zero-One Matrices. Cambridge University Press.

Dwyer, Paul S. 1967. â€śSome Applications of Matrix Derivatives in Multivariate Analysis.â€ť Journal of the American Statistical Association 62 (318): 607. https://doi.org/10.2307/2283988.

Gene H. Golub, and Charles F. van Loan. 1983. Matrix Computations. JHU Press.

George A. F. Seber. 2007. A Matrix Handbook for Statisticians. Wiley.

Giles, M. 2008. â€śAn Extended Collection of Matrix Derivative Results for Forward and Reverse Mode Automatic Differentiation.â€ť Http://Eprints.maths.ox.ac.uk/1079, January. http://www2.maths.ox.ac.uk/~gilesm/files/NA-08-01.pdf.

Giles, Mike B. 2008. â€śCollected Matrix Derivative Results for Forward and Reverse Mode Algorithmic Differentiation.â€ť In Advances in Automatic Differentiation, edited by Christian H. Bischof, H. Martin BĂĽcker, Paul Hovland, Uwe Naumann, and Jean Utke, 64:35â€“44. Berlin, Heidelberg: Springer Berlin Heidelberg. http://eprints.maths.ox.ac.uk/1079/.

Laue, Soeren, Matthias Mitterreiter, and Joachim Giesen. 2018. â€śComputing Higher Order Derivatives of Matrix and Tensor Expressions.â€ť In Advances in Neural Information Processing Systems 31, edited by S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, 2750â€“9. Curran Associates, Inc. http://papers.nips.cc/paper/7540-computing-higher-order-derivatives-of-matrix-and-tensor-expressions.pdf.

Magnus, Jan R., and Heinz Neudecker. 1999. Matrix Differential Calculus with Applications in Statistics and Econometrics. Rev. ed. New York: John Wiley. http://www.janmagnus.nl/misc/mdc2007-3rdedition.

Minka, Thomas P. 2000. â€śOld and New Matrix Algebra Useful for Statistics.â€ť http://msr-waypoint.com/en-us/um/people/minka/papers/matrix/minka-matrix.pdf.

Parlett, Beresford N. 2000. â€śThe QR Algorithm.â€ť Computing in Science & Engineering 2 (1): 38â€“42. https://doi.org/10.1109/5992.814656.

Petersen, Kaare Brandt, and Michael Syskind Pedersen. 2012. â€śThe Matrix Cookbook.â€ť http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=3274.

Willi-Hans Steeb. 2006. Problems and Solutions in Introductory and Advanced Matrix Calculus. World Scientific.