Frequentist model selection is presumably not the only type. But what is model selection in a Bayesian context? Surely you don’t ever get some models with zero posterior proability? I thought we just kept all the models and weighted by posterior likelihood when making predictions? I still might wish to discard some models for reasons of computational tractability or what-have-you? What price do I pay for what sounds like unorthodoxy by the standards of my Intro-to-Bayes class?
Interesting special case: Bayesian sparsity.
Cross-validation and Bayes
There is a relation between cross-validation and Bayes evidence, a.k.a. marginal likelihood - see FoHo19.
- LiDu16: Meng Li, David B. Dunson (2016) A framework for probabilistic inferences from imperfect models. ArXiv:1611.01241 [Stat].
- Stei08: Michael L. Stein (2008) A modeling approach for large spatial datasets. Journal of the Korean Statistical Society, 37(1), 3–10. DOI
- VeOj12: Aki Vehtari, Janne Ojanen (2012) A survey of Bayesian predictive methods for model assessment, selection and comparison. Statistics Surveys, 6, 142–228. DOI
- OSYR17: John T. Ormerod, Michael Stewart, Weichang Yu, Sarah E. Romanes (2017) Bayesian hypothesis tests with diffuse priors: Can we have our cake and eat it too? ArXiv:1710.09146 [Math, Stat].
- Efro12: Bradley Efron (2012) Bayesian inference and the parametric bootstrap. The Annals of Applied Statistics, 6(4), 1971–1997. DOI
- Mack99: David JC MacKay (1999) Comparison of approximate methods for handling hyperparameters. Neural Computation, 11(5), 1035–1068. DOI
- PiVe17: Juho Piironen, Aki Vehtari (2017) Comparison of Bayesian predictive methods for model selection. Statistics and Computing, 27(3), 711–735. DOI
- KaLa04: Joseph B. Kadane, Nicole A. Lazar (2004) Methods and criteria for model selection. Journal of the American Statistical Association, 99(465), 279–290. DOI
- ClHj08: Gerda Claeskens, Nils Lid Hjort (2008) Model selection and model averaging. Cambridge ; New York: Cambridge University Press
- FoHo19: Edwin Fong, Chris Holmes (2019) On the marginal likelihood and cross-validation. ArXiv:1905.08737 [Stat].
- Mack95: David J. C. Mackay (1995) Probable networks and plausible predictions — a review of practical Bayesian methods for supervised neural networks. Network: Computation in Neural Systems, 6(3), 469–505. DOI
- IsRa05: Hemant Ishwaran, J. Sunil Rao (2005) Spike and slab variable selection: Frequentist and Bayesian strategies. The Annals of Statistics, 33(2), 730–773. DOI
- CGML01: Hugh Chipman, Edward I. George, Robert E. McCulloch, P Lahiri (2001) The Practical Implementation of Bayesian Model Selection. In Model selection (Vol. 38). Beachwood, OH: Institute of Mathematical Statistics DOI
- RoGe18: Veronika Ročková, Edward I. George (2018) The Spike-and-Slab LASSO. Journal of the American Statistical Association, 113(521), 431–444. DOI