Inverse-Wishart distribution

If ${\displaystyle {\mathbf {X} }\sim {\mathcal {W}}({\mathbf {\Sigma } },\nu )}$ and ${\displaystyle {\mathbf {\Sigma } }}$ is of size ${\displaystyle p\times p}$, then ${\displaystyle \mathbf {A} ={\mathbf {X} }^{-1}}$ has an inverse Wishart distribution ${\displaystyle \mathbf {A} \sim {\mathcal {W}}^{-1}({\mathbf {\Sigma } }^{-1},\nu )}$ .[4]

Suppose ${\displaystyle {\mathbf {A} }\sim {\mathcal {W}}^{-1}({\mathbf {\Psi } },\nu )}$ has an inverse Wishart distribution. Partition the matrices ${\displaystyle {\mathbf {A} }}$ and ${\displaystyle {\mathbf {\Psi } }}$ conformably with each other

${\displaystyle {\mathbf {A} }={\begin{bmatrix}\mathbf {A} _{11}&\mathbf {A} _{12}\\\mathbf {A} _{21}&\mathbf {A} _{22}\end{bmatrix}},\;{\mathbf {\Psi } }={\begin{bmatrix}\mathbf {\Psi } _{11}&\mathbf {\Psi } _{12}\\\mathbf {\Psi } _{21}&\mathbf {\Psi } _{22}\end{bmatrix}}}$

where ${\displaystyle {\mathbf {A} _{ij}}}$ and ${\displaystyle {\mathbf {\Psi } _{ij}}}$ are ${\displaystyle p_{i}\times p_{j}}$ matrices, then we have

1. ${\displaystyle \mathbf {A} _{11}}$ is independent of ${\displaystyle \mathbf {A} _{11}^{-1}\mathbf {A} _{12}}$ and ${\displaystyle {\mathbf {A} }_{22\cdot 1}}$, where ${\displaystyle {\mathbf {A} _{22\cdot 1}}={\mathbf {A} }_{22}-{\mathbf {A} }_{21}{\mathbf {A} }_{11}^{-1}{\mathbf {A} }_{12}}$ is the Schur complement of ${\displaystyle {\mathbf {A} _{11}}}$ in ${\displaystyle {\mathbf {A} }}$;
2. ${\displaystyle {\mathbf {A} _{11}}\sim {\mathcal {W}}^{-1}({\mathbf {\Psi } _{11}},\nu -p_{2})}$;
3. ${\displaystyle {\mathbf {A} }_{11}^{-1}{\mathbf {A} }_{12}\mid {\mathbf {A} }_{22\cdot 1}\sim MN_{p_{1}\times p_{2}}({\mathbf {\Psi } }_{11}^{-1}{\mathbf {\Psi } }_{12},{\mathbf {A} }_{22\cdot 1}\otimes {\mathbf {\Psi } }_{11}^{-1})}$, where ${\displaystyle MN_{p\times q}(\cdot ,\cdot )}$ is a matrix normal distribution;
4. ${\displaystyle {\mathbf {A} }_{22\cdot 1}\sim {\mathcal {W}}^{-1}({\mathbf {\Psi } }_{22\cdot 1},\nu )}$, where ${\displaystyle {\mathbf {\Psi } _{22\cdot 1}}={\mathbf {\Psi } }_{22}-{\mathbf {\Psi } }_{21}{\mathbf {\Psi } }_{11}^{-1}{\mathbf {\Psi } }_{12}}$;

Suppose we wish to make inference about a covariance matrix ${\displaystyle {\mathbf {\Sigma } }}$ whose prior ${\displaystyle {p(\mathbf {\Sigma } )}}$ has a ${\displaystyle {\mathcal {W}}^{-1}({\mathbf {\Psi } },\nu )}$ distribution. If the observations ${\displaystyle \mathbf {X} =[\mathbf {x} _{1},\ldots ,\mathbf {x} _{n}]}$ are independent p-variate Gaussian variables drawn from a ${\displaystyle N(\mathbf {0} ,{\mathbf {\Sigma } })}$ distribution, then the conditional distribution ${\displaystyle {p(\mathbf {\Sigma } \mid \mathbf {X} )}}$ has a ${\displaystyle {\mathcal {W}}^{-1}({\mathbf {A} }+{\mathbf {\Psi } },n+\nu )}$ distribution, where ${\displaystyle {\mathbf {A} }=\mathbf {X} \mathbf {X} ^{T}}$.

Because the prior and posterior distributions are the same family, we say the inverse Wishart distribution is conjugate to the multivariate Gaussian.

Due to its conjugacy to the multivariate Gaussian, it is possible to marginalize out (integrate out) the Gaussian's parameter ${\displaystyle \mathbf {\Sigma } }$, using the formula ${\displaystyle p(x)={\frac {p(x|\Sigma )p(\Sigma )}{p(\Sigma |x)}}}$ and the linear algebra identity ${\displaystyle v^{T}\Omega v={\text{tr}}(\Omega vv^{T})}$:

${\displaystyle f_{\mathbf {X} \,\mid \,\Psi ,\nu }(\mathbf {x} )=\int f_{\mathbf {X} \,\mid \,\mathbf {\Sigma } \,=\,\sigma }(\mathbf {x} )f_{\mathbf {\Sigma } \,\mid \,\mathbf {\Psi } ,\nu }(\sigma )\,d\sigma ={\frac {|\mathbf {\Psi } |^{\nu /2}\Gamma _{p}\left({\frac {\nu +n}{2}}\right)}{\pi ^{np/2}|\mathbf {\Psi } +\mathbf {A} |^{(\nu +n)/2}\Gamma _{p}({\frac {\nu }{2}})}}}$

(this is useful because the variance matrix ${\displaystyle \mathbf {\Sigma } }$ is not known in practice, but because ${\displaystyle {\mathbf {\Psi } }}$ is known a priori, and ${\displaystyle {\mathbf {A} }}$ can be obtained from the data, the right hand side can be evaluated directly). The inverse-Wishart distribution as a prior can be constructed via existing transferred prior knowledge.[5]

The following is based on Press, S. J. (1982) "Applied Multivariate Analysis", 2nd ed. (Dover Publications, New York), after reparameterizing the degree of freedom to be consistent with the p.d.f. definition above.

Let ${\displaystyle W\sim {\mathcal {W}}(\mathbf {\Psi } ^{-1},\nu )}$ with ${\displaystyle \nu \geq p}$ and ${\displaystyle X\doteq W^{-1}}$, so that ${\displaystyle X\sim {\mathcal {W}}^{-1}(\mathbf {\Psi } ,\nu )}$.

The mean:[4]^: 85

${\displaystyle \operatorname {E} (\mathbf {X} )={\frac {\mathbf {\Psi } }{\nu -p-1}}.}$

The variance of each element of ${\displaystyle \mathbf {X} }$:

${\displaystyle \operatorname {Var} (x_{ij})={\frac {(\nu -p+1)\psi _{ij}^{2}+(\nu -p-1)\psi _{ii}\psi _{jj}}{(\nu -p)(\nu -p-1)^{2}(\nu -p-3)}}}$

The variance of the diagonal uses the same formula as above with ${\displaystyle i=j}$, which simplifies to:

${\displaystyle \operatorname {Var} (x_{ii})={\frac {2\psi _{ii}^{2}}{(\nu -p-1)^{2}(\nu -p-3)}}.}$

The covariance of elements of ${\displaystyle \mathbf {X} }$ are given by:

${\displaystyle \operatorname {Cov} (x_{ij},x_{k\ell })={\frac {2\psi _{ij}\psi _{k\ell }+(\nu -p-1)(\psi _{ik}\psi _{j\ell }+\psi _{i\ell }\psi _{kj})}{(\nu -p)(\nu -p-1)^{2}(\nu -p-3)}}}$

The same results are expressed in Kronecker product form by von Rosen[6] as follows:

${\displaystyle {\begin{aligned}\mathbf {E} \left(W^{-1}\otimes W^{-1}\right)&=c_{1}\Psi \otimes \Psi +c_{2}Vec(\Psi )Vec(\Psi )^{T}+c_{2}K_{pp}\Psi \otimes \Psi \\\mathbf {Cov} _{\otimes }\left(W^{-1},W^{-1}\right)&=(c_{1}-c_{3})\Psi \otimes \Psi +c_{2}Vec(\Psi )Vec(\Psi )^{T}+c_{2}K_{pp}\Psi \otimes \Psi \end{aligned}}}$

where

${\displaystyle {\begin{aligned}c_{2}&=\left[(\nu -p)(\nu -p-1)(\nu -p-3)\right]^{-1}\\c_{1}&=(\nu -p-2)c_{2}\\c_{3}&=(\nu -p-1)^{-2},\end{aligned}}}$

${\displaystyle K_{pp}{\text{ is a }}p^{2}\times p^{2}}$ commutation matrix

${\displaystyle \mathbf {Cov} _{\otimes }\left(W^{-1},W^{-1}\right)=\mathbf {E} \left(W^{-1}\otimes W^{-1}\right)-\mathbf {E} \left(W^{-1}\right)\otimes \mathbf {E} \left(W^{-1}\right).}$

There appears to be a typo in the paper whereby the coefficient of ${\displaystyle K_{pp}\Psi \otimes \Psi }$ is given as ${\displaystyle c_{1}}$ rather than ${\displaystyle c_{2}}$, and that the expression for the mean square inverse Wishart, corollary 3.1, should read

${\displaystyle \mathbf {E} \left[W^{-1}W^{-1}\right]=(c_{1}+c_{2})\Sigma ^{-1}\Sigma ^{-1}+c_{2}\Sigma ^{-1}\mathbf {tr} (\Sigma ^{-1}).}$

To show how the interacting terms become sparse when the covariance is diagonal, let ${\displaystyle \Psi =\mathbf {I} _{3\times 3}}$ and introduce some arbitrary parameters ${\displaystyle u,v,w}$:

${\displaystyle \mathbf {E} \left(W^{-1}\otimes W^{-1}\right)=u\Psi \otimes \Psi +v\,\mathrm {vec} (\Psi )\,\mathrm {vec} (\Psi )^{T}+wK_{pp}\Psi \otimes \Psi .}$

where ${\displaystyle \mathrm {vec} }$ denotes the matrix vectorization operator. Then the second moment matrix becomes

${\displaystyle \mathbf {E} \left(W^{-1}\otimes W^{-1}\right)={\begin{bmatrix}u+v+w&\cdot &\cdot &\cdot &v&\cdot &\cdot &\cdot &v\\\cdot &u&\cdot &w&\cdot &\cdot &\cdot &\cdot &\cdot \\\cdot &\cdot &u&\cdot &\cdot &\cdot &w&\cdot &\cdot \\\cdot &w&\cdot &u&\cdot &\cdot &\cdot &\cdot &\cdot \\v&\cdot &\cdot &\cdot &u+v+w&\cdot &\cdot &\cdot &v\\\cdot &\cdot &\cdot &\cdot &\cdot &u&\cdot &w&\cdot \\\cdot &\cdot &w&\cdot &\cdot &\cdot &u&\cdot &\cdot \\\cdot &\cdot &\cdot &\cdot &\cdot &w&\cdot &u&\cdot \\v&\cdot &\cdot &\cdot &v&\cdot &\cdot &\cdot &u+v+w\\\end{bmatrix}}}$

which is non-zero only when involving the correlations of diagonal elements of ${\displaystyle W^{-1}}$, all other elements are mutually uncorrelated, though not necessarily statistically independent. The variances of the Wishart product are also obtained by Cook et al.[7] in the singular case and, by extension, to the full rank case.

Muirhead[8] shows in Theorem 3.2.5 that if ${\displaystyle A}$ is distributed as ${\displaystyle {\mathcal {W}}_{m}(n,\Sigma )}$ and ${\displaystyle V}$ is a random vector, independent of ${\displaystyle A}$, then ${\displaystyle {\frac {V^{T}\Sigma ^{-1}V}{V^{T}A^{-1}V}}\sim \chi _{n-m+1}^{2}}$ and it follows that ${\displaystyle {\frac {V^{T}A^{-1}V}{V^{T}\Sigma ^{-1}V}}}$ follows an Inverse-chi-squared distribution. Setting ${\displaystyle V=(1,\,0,\cdots ,0)^{T}}$ the marginal distribution of the leading diagonal element is thus

${\displaystyle {\frac {[A^{-1}]_{1,1}}{[\Sigma ^{-1}]_{1,1}}}\sim {\text{Inv-}}\chi ^{2}(n-m+1)={\frac {2^{-k/2}}{\Gamma (k/2)}}x^{-k/2-1}e^{-1/(2x)},\;\;k=n-m+1}$

and by rotating ${\displaystyle V}$ end-around a similar result applies to all diagonal elements ${\displaystyle [A^{-1}]_{i,i}}$.

A corresponding result in the complex Wishart case was shown by Brennan and Reed[9] and the uncorrelated inverse complex Wishart ${\displaystyle {\mathcal {W_{C}}}^{-1}(\mathbf {I} ,\nu ,p)}$ was shown by Shaman[10] to have diagonal statistical structure in which the leading diagonal elements are correlated, while all other element are uncorrelated.