The Living Thing / Notebooks :

Model averaging

On keeping many incorrect hypotheses and using them all as one goodish one

A mere placeholder to remind me to create a model averaging notebook, since I’ve seen the idea pop up in disconnected areas recently, specifically a Bayesian heuristic for dropout in neural nets, AIC for frequentist model averaging, and in a statistical learning context for optimal time series prediction.

Relationship to Bayesian posterior distributions?

This seems to not be quite the same thing as bagging - or is it?

Model weights are often in terms of degrees-of-freedom penalties. It would probably be an instructive exercise for me to work out why for myself.


Bates, J. M., & Granger, C. W. J.(1969) The Combination of Forecasts. Journal of the Operational Research Society, 20(4), 451–468. DOI.
Buckland, S. T., Burnham, K. P., & Augustin, N. H.(1997) Model Selection: An Integral Part of Inference. Biometrics, 53(2), 603–618. DOI.
Claeskens, G., & Hjort, N. L.(2008) Model selection and model averaging. . Cambridge ; New York: Cambridge University Press
Clyde, M., & George, E. I.(2004) Model Uncertainty. Statistical Science, 19(1), 81–94. DOI.
Fragoso, T. M., & Neto, F. L.(2015) Bayesian model averaging: A systematic review and conceptual classification. ArXiv:1509.08864 [Stat].
Hansen, B. E.(2007) Least Squares Model Averaging. Econometrica, 75(4), 1175–1189. DOI.
Hjort, N. L., & Claeskens, G. (2003) Frequentist Model Average Estimators. Journal of the American Statistical Association, 98(464), 879–899. DOI.
Hoeting, J. A., Madigan, D., Raftery, A. E., & Volinsky, C. T.(1999) Bayesian model averaging: a tutorial. Statistical Science, 14(4), 382–417. DOI.
Leung, G., & Barron, A. R.(2006) Information Theory and Mixing Least-Squares Regressions. IEEE Transactions on Information Theory, 52(8), 3396–3410. DOI.
Phillips, R. F.(1987) Composite Forecasting: An Integrated Approach and Optimality Reconsidered. Journal of Business & Economic Statistics, 5(3), 389–395. DOI.
Shen, X., & Huang, H.-C. (2006) Optimal Model Assessment, Selection, and Combination. Journal of the American Statistical Association, 101(474), 554–568. DOI.
Wang, H., Zhang, X., & Zou, G. (2009) Frequentist model averaging estimation: a review. Journal of Systems Science and Complexity, 22(4), 732. DOI.
Zhang, X., & Liang, H. (2011) Focused information criterion and model averaging for generalized additive partial linear models. The Annals of Statistics, 39(1), 174–200. DOI.