The Living Thing / Notebooks :

Effective sample size


When your experiment design (e.g. because it is a time series, or because of non-random sampling is highly correlated, your data might give you less information than you hope, or expect from the uncorrelated case.

This is a kind of dual to effective degrees of freedom, which tells you how far your sample size can get you.

Turns out to be important in, e.g. LASSO and, circularly, covariance estimation.

Huber’s (1981) “equivalent number of observations” is probably the same?

Monte Carlo estimation

Related, but a different definition. Not about experimental samples, but number of simulations in simulation-based inference where you are using importance sampling. e.g. Sebastian Nowozin, Effective Sample Size in Importance Sampling.

[The effective sample size] can be used after or during importance sampling to provide a quantitative measure of the quality of the estimated mean. Even better, the estimate is provided on a natural scale of worth in samples from p, that is, if we use (n=1000$ samples $X_iq) and obtain an ESS of say 350 then this indicates that the quality of our estimate is about the same as if we would have used 350 direct samples[…]

If you maximize effective sample size with your design, you are doing importance sampling, I guess? Sebastian Nowozin makes this clear in excellent blog post.