Spike-and-slab regression
Spike-and-slab regression is a type of Bayesian linear regression in which a particular hierarchical prior distribution for the regression coefficients is chosen such that only a subset of the possible regressors is retained. The technique is particularly useful when the number of possible predictors is larger than the number of observations.[1] The idea of the spike-and-slab model was originally proposed by Mitchell & Beauchamp (1988).[2] The approach was further significantly developed by Madigan & Raftery (1994)[3] and George & McCulloch (1997).[4] A recent and important contribution to this literature is Ishwaran & Rao (2005).[5]
Model description
Suppose we have P possible predictors in some model. Vector γ has a length equal to P and consists of zeros and ones. This vector indicates whether a particular variable is included in the regression or not. If no specific prior information on initial inclusion probabilities of particular variables is available, a Bernoulli prior distribution is a common default choice.[6] Conditional on a predictor being in the regression, we identify a prior distribution for the model coefficient, which corresponds to that variable (β). A common choice on that step is to use a normal prior with a mean equal to zero and a large variance calculated based on (where is a design matrix of explanatory variables of the model).[7]
A draw of γ from its prior distribution is a list of the variables included in the regression. Conditional on this set of selected variables, we take a draw from the prior distribution of the regression coefficients (if γi = 1 then βi ≠ 0 and if γi = 0 then βi = 0). βγ denotes the subset of β for which γi = 1. In the next step, we calculate a posterior probability for both inclusion and coefficients by applying a standard statistical procedure.[8] All steps of the described algorithm are repeated thousands of times using the Markov chain Monte Carlo (MCMC) technique. As a result, we obtain a posterior distribution of γ (variable inclusion in the model), β (regression coefficient values) and the corresponding prediction of y.
The model got its name (spike-and-slab) due to the shape of the two prior distributions. The "spike" is the probability of a particular coefficient in the model to be zero. The "slab" is the prior distribution for the regression coefficient values.
An advantage of Bayesian variable selection techniques is that they are able to make use of prior knowledge about the model. In the absence of such knowledge, some reasonable default values can be used; to quote Scott and Varian (2013): "For the analyst who prefers simplicity at the cost of some reasonable assumptions, useful prior information can be reduced to an expected model size, an expected R2, and a sample size ν determining the weight given to the guess at R2."[6] Some researchers suggest the following default values: R2 = 0.5, ν = 0.01, and π = 0.5 (parameter of a prior Bernoulli distribution).[6]
References
- Varian, Hal R. (2014). "Big Data: New Tricks for Econometrics". Journal of Economic Perspectives. 28 (2): 3–28. doi:10.1257/jep.28.2.3.
- Mitchell, T. J.; Beauchamp, J. J. (1988). "Bayesian Variable Selection in Linear Regression". Journal of the American Statistical Association. 83 (404): 1023–1032. doi:10.1080/01621459.1988.10478694.
- Madigan, David; Raftery, Adrian E. (1994). "Model Selection and Accounting for Model Uncertainty in Graphical Models Using Occam's Window". Journal of the American Statistical Association. 89 (428): 1535–1546. doi:10.1080/01621459.1994.10476894.
- George, Edward I.; McCulloch, Robert E. (1997). "Approaches for Bayesian Variable Selection". Statistica Sinica. 7 (2): 339–373. JSTOR 24306083.
- Ishwaran, Hemant; Rao, J. Sunil (2005). "Spike and slab variable selection: frequentist and Bayesian strategies". The Annals of Statistics. 33 (2): 730–773. arXiv:math/0505633. Bibcode:2005math......5633I. doi:10.1214/009053604000001147. S2CID 9004248.
- Scott, Steven L.; Varian, Hal R. (2014). "Predicting the Present with Bayesian Structural Time Series". International Journal of Mathematical Modelling and Numerical Optimisation. 5 (1–2): 4–23. doi:10.1504/IJMMNO.2014.059942.
- "Bayesian variable selection for nowcasting economic time series" (PDF).
- Brodersen, Kay H.; Gallusser, Fabian; Koehler, Jim; Remy, Nicolas; Scott, Steven L. (2015). "Inferring causal impact using Bayesian structural time-series models". Annals of Applied Statistics. 9: 247–274. arXiv:1506.00356. doi:10.1214/14-AOAS788. S2CID 2879370.
Further reading
- Congdon, Peter D. (2020). "Regression Techniques using Hierarchical Priors". Bayesian Hierarchical Models (2nd ed.). Boca Raton: CRC Press. pp. 253–315. ISBN 978-1-03-217715-1.