The Living Thing / Notebooks : Feedback systems

Stuff I want to learn about signal processing with feedback.

z-Transforms, filter design etc.

I am going to consider this in discrete time (i.e. for digital implementation) unless otherwise stated.

TBD: explain these.

This notebook is about designing properties of systems to given specifications. For inference from data, you want system identification; and for prediction, you want state filters.

Related, musical: delays and reverbs.

In the linear case, this is well studied, e.g. we all had to play with ARMA models at some point.

More generally, we want to handle arbitrary Box-Jenkins models; ARMA, ARIMA, ARMAX, BANANARAMA and so on.

If you make the assumption that the system of interest is linear this might be at least a starting point for a more plausible model and you have lots of very convenient estimation theory. Your system of interest is possibly locally linear, so if you don’t think about it too hard, making this a popular choice for the not-thinking-about-it-too-hard phase of research. They fit very easily AIC model selection/averaging and maximum likelihood frameworks.

Anyway, where you know that the dynamics are globally nearly linear (say, you are modeling the propagation of waves through the air where physics tells you the non-quite linear-longitudinal pressure wave can be approximated by a linear transverse wave over a wide range of pressures and frequencies), this might get you all the way. Usual linear model caveats apply.

Quick and dirty digital filter design

Julius O. Smith III’s lovingly curated encyclopaedia of filter tricks covers everything commonly used in audio, at the cost of eyeball-searing ugliness and impenetrable curtness. If you already know some linear systems theory, that is a great way of learning new tricks, or re-understanding old tricks.

Biquad filter design:

The classic, ancient musicdsp cookbook by Robert Bristow-Johnson.

See also the Earlevel biquad formulae crib sheet by Nigel Redmon, and also his digital SVF intro.

Relationship of discrete LTI to continuous time processes

Martin (Mart99):

Consider the basic autoregressive model,

\begin{equation*} Y(k) = \sum_{j=1}^pa_jY(k-j)=\epsilon(k). \end{equation*}

Estimating AR(p) coefficients:

The [power] spectrum is easily obtained from [the above] as

\begin{align*} P(f) = \frac{\sigma^2}{|1+ \sum_{j=1}^pa_jz^{-1}|^2},\\ z=\exp 2\pi if\delta t \end{align*}

with \(\delta t\) the intersample spacing.[…] for any given set of data, we need to be able to estimate the AR coeficients \(\{a_j\}_{j=1}^N\) conveniently. Three methods for achieving this are the Yule-Walker, Burg and Covariance methods. The Yule-Walker technique uses the sample autocovariance to obtain the coefficients; the Covariance method defines, for a set of numbers \(\mathbf{a}=\{a_j\}_{j=1}^N,\) a quantity known as the total forward and backward prediction error power:

\begin{equation*} E(Y,\mathbf{a}) = \frac{1}{2(N-p)}\sum_{n=p+1}^N\left\{ \left|Y(n)+\sum{j=1}^pa_jY(n-p)\right|^2 + \left|Y(n-p)+\sum{j=1}^pa^*_jY(n-p+j)\right|^2 \right\} \end{equation*}

and minimises this w.r.t. \(\mathbf{a}\). As \(E(Y, \mathbf{a})\) is a quadratic function of \(\mathbf{a}\), \(\partial E(Y, \mathbf{a})/partial a\) is linear in \(\mathbf{a}\) and so this is a linear optimisation problem. The Burg method is a constrained minimisation of \(E(Y, \mathbf{a})\) using the Levinson recursion, a computational device derived from the Yule-Walker method.


Feedback Systems: An Introduction for Scientists and Engineers by Karl J. Åström and Richard M. Murray is a really interesting control systems theory course from Caltech; The first half of the course touches on a lot of this.

They maintain a supporting python toolkit, python-control.

Signal Processing for Communications by Paolo Prandoni and Martin Vetterli is available online. Vetterli is good.

Antoniou has been generally recommended if you want to get hands-on ASAP. (Anto05)

However, even quicker is the musicdsp biquad filter cookbook.

Lucid introduction: dsprelated’s state space filter tutorial

Orfandis’ textbook is revised and free online. (Orfa96)

Oppenheim and Verghese, SIGNALS, SYSTEMS, and INFERENCE is free online.

Refs, sundry

Antoniou, A. (2005) Digital signal processing: signals, systems and filters. . New York: McGraw-Hill
Bartlett, M. S.(1946) On the Theoretical Specification and Sampling Properties of Autocorrelated Time-Series. Supplement to the Journal of the Royal Statistical Society, 8(1), 27–41. DOI.
Box, G. E. P., Jenkins, G. M., Reinsel, G. C., & Ljung, G. M.(2016) Time series analysis: forecasting and control. (Fifth edition.). Hoboken, New Jersey: John Wiley & Sons, Inc
Hencic, A., & Gouriéroux, C. (2015) Noncausal Autoregressive Model in Application to Bitcoin/USD Exchange Rates. In V.-N. Huynh, V. Kreinovich, S. Sriboonchitta, & K. Suriya (Eds.), Econometrics of Risk (pp. 17–40). Springer International Publishing DOI.
Holan, S. H., Lund, R., & Davis, G. (2010) The ARMA alphabet soup: A tour of ARMA model variants. Statistics Surveys, 4, 232–274. DOI.
Marple, S. L., Jr. (1987) Digital spectral analysis with applications.
Martin, R. J.(1999) Autoregression and irregular sampling: Spectral estimation. Signal Processing, 77(2), 139–157. DOI.
Moon, T. K., & Stirling, W. C.(2000) Mathematical methods and algorithms for signal processing. . Upper Saddle River, NJ: Prentice Hall
Oppenheim, A. V., Schafer, R. W., & Buck, J. R.(1999) Discrete-time signal processing. (2nd ed.). Upper Saddle River, N.J: Prentice Hall
Orfanidis, S. J.(1996) Introduction to signal processing. . Englewood Cliffs, N.J: Prentice Hall
Prandoni, P., & Vetterli, M. (2008) Signal processing for communications. . Lausanne: EPFL Press
Scargle, J. D.(1981) Studies in astronomical time series analysis I-Modeling random processes in the time domain. The Astrophysical Journal Supplement Series, 45, 1–71.
Stoica, P., & Moses, R. L.(2005) Spectral Analysis of Signals. (1 edition.). Upper Saddle River, N.J: Prentice Hall
Therrien, C. W.(1992) Discrete random signals and statistical signal processing. . Englewood Cliffs, NJ: Prentice Hall