What if you like the flavours of both Bayesian inference and the implicit model selection of sparse inference? Can you cook Bayesian-Frequentist fusion cuisine with this novelty ingredient?
Laplace priors on linear regression coefficients, includes normal lasso as a MAP estimate. Pro: very easy to derive frequentist LASSO as a MAP estimate from this prior. Con: Not actually sparse for non-MAP uses. I have no need for this right now, but I did enjoy Dan Simpson’s critique.
Stan guy, Michael Betancourt introduces some issues with LASSO-type inference for Bayesians with a slant towards Horseshoe-type priors in preference spike and slab, possibly because hierarchical mixtures like spike-and-slab are not that great in Stan, albeit possible.
- GeMc97: Edward I. George, Robert McCulloch (1997) Approaches for bayesian variable selection. Statistica Sinica, 7(2), 339–373.
- Ročk18: Veronika Ročková (2018) Bayesian estimation of sparse signals with a continuous spike-and-slab prior. The Annals of Statistics, 46(1), 401–437. DOI
- MiBe88: T. J. Mitchell, J. J. Beauchamp (1988) Bayesian Variable Selection in Linear Regression. Journal of the American Statistical Association, 83(404), 1023–1032. DOI
- CaPS09: Carlos M. Carvalho, Nicholas G. Polson, James G. Scott (2009) Handling Sparsity via the Horseshoe. In Artificial Intelligence and Statistics (pp. 73–80).
- BGKR15: Kay H. Brodersen, Fabian Gallusser, Jim Koehler, Nicolas Remy, Steven L. Scott (2015) Inferring causal impact using Bayesian structural time-series models. The Annals of Applied Statistics, 9(1), 247–274. DOI
- MaRa94: David Madigan, Adrian E. Raftery (1994) Model Selection and Accounting for Model Uncertainty in Graphical Models Using Occam’s Window. Journal of the American Statistical Association, 89(428), 1535–1546. DOI
- SmKo96: Michael Smith, Robert Kohn (1996) Nonparametric regression using Bayesian variable selection. Journal of Econometrics, 75(2), 317–343. DOI
- ScVa13: Steven L. Scott, Hal R. Varian (2013) Predicting the Present with Bayesian Structural Time Series (SSRN Scholarly Paper No. ID 2304426). Rochester, NY: Social Science Research Network
- PiVe17: Juho Piironen, Aki Vehtari (2017) Sparsity information and regularization in the horseshoe and other shrinkage priors. Electronic Journal of Statistics, 11(2), 5018–5051. DOI
- IsRa05: Hemant Ishwaran, J. Sunil Rao (2005) Spike and slab variable selection: Frequentist and Bayesian strategies. The Annals of Statistics, 33(2), 730–773. DOI
- TiLá11: Michalis K. Titsias, Miguel Lázaro-Gredilla (2011) Spike and Slab Variational Inference for Multi-Task and Multiple Kernel Learning. In Advances in Neural Information Processing Systems 24 (pp. 2339–2347). Curran Associates, Inc.
- CaPS10: Carlos M. Carvalho, Nicholas G. Polson, James G. Scott (2010) The horseshoe estimator for sparse signals. Biometrika, 97(2), 465–480. DOI
- RoGe18: Veronika Ročková, Edward I. George (2018) The Spike-and-Slab LASSO. Journal of the American Statistical Association, 113(521), 431–444. DOI