Marginal likelihood

Given a set of independent identically distributed data points ${\displaystyle \mathbf {X} =(x_{1},\ldots ,x_{n}),}$ where ${\displaystyle x_{i}\sim p(x|\theta )}$ according to some probability distribution parameterized by ${\displaystyle \theta }$, where ${\displaystyle \theta }$ itself is a random variable described by a distribution, i.e. ${\displaystyle \theta \sim p(\theta \mid \alpha ),}$ the marginal likelihood in general asks what the probability ${\displaystyle p(\mathbf {X} \mid \alpha )}$ is, where ${\displaystyle \theta }$ has been marginalized out (integrated out):

${\displaystyle p(\mathbf {X} \mid \alpha )=\int _{\theta }p(\mathbf {X} \mid \theta )\,p(\theta \mid \alpha )\ \operatorname {d} \!\theta }$

The above definition is phrased in the context of Bayesian statistics in which case ${\displaystyle p(\theta \mid \alpha )}$ is called prior density and ${\displaystyle p(\mathbf {X} \mid \theta )}$ is the likelihood. The marginal likelihood quantifies the agreement between data and prior in a geometric sense made precise in de Carvalho et al. (2019). In classical (frequentist) statistics, the concept of marginal likelihood occurs instead in the context of a joint parameter ${\displaystyle \theta =(\psi ,\lambda )}$, where ${\displaystyle \psi }$ is the actual parameter of interest, and ${\displaystyle \lambda }$ is a non-interesting nuisance parameter. If there exists a probability distribution for ${\displaystyle \lambda }$, it is often desirable to consider the likelihood function only in terms of ${\displaystyle \psi }$, by marginalizing out ${\displaystyle \lambda }$:

${\displaystyle {\mathcal {L}}(\psi$ ;\mathbf {X} )=p(\mathbf {X} \mid \psi )=\int _{\lambda }p(\mathbf {X} \mid \lambda ,\psi )\,p(\lambda \mid \psi )\ \operatorname {d} \!\lambda }

Unfortunately, marginal likelihoods are generally difficult to compute. Exact solutions are known for a small class of distributions, particularly when the marginalized-out parameter is the conjugate prior of the distribution of the data. In other cases, some kind of numerical integration method is needed, either a general method such as Gaussian integration or a Monte Carlo method, or a method specialized to statistical problems such as the Laplace approximation, Gibbs/Metropolis sampling, or the EM algorithm.

It is also possible to apply the above considerations to a single random variable (data point) ${\displaystyle x}$, rather than a set of observations. In a Bayesian context, this is equivalent to the prior predictive distribution of a data point.

• Empirical Bayes methods
• Lindley's paradox
• Marginal probability
• Bayesian information criterion

• Charles S. Bos. "A comparison of marginal likelihood computation methods". In W. Härdle and B. Ronz, editors, COMPSTAT 2002: Proceedings in Computational Statistics, pp. 111–117. 2002. (Available as a preprint on the web: )
• de Carvalho, Miguel; Page, Garritt; Barney, Bradley (2019). "On the geometry of Bayesian inference". Bayesian Analysis. 14 (4): 1013‒1036. (Available as a preprint on the web: )
• Lambert, Ben (2018). "The devil is in the denominator". A Student's Guide to Bayesian Statistics. Sage. pp. 109–120. ISBN 978-1-4739-1636-4.
• The on-line textbook: Information Theory, Inference, and Learning Algorithms, by David J.C. MacKay.