Holm–Bonferroni method

The simple Bonferroni correction rejects only null hypotheses with p-value less than ${\displaystyle {\frac {\alpha }{m}}}$, in order to ensure that the FWER, i.e., the risk of rejecting one or more true null hypotheses (i.e., of committing one or more type I errors) is at most ${\displaystyle \alpha }$. The cost of this protection against type I errors is an increased risk of failing to reject one or more false null hypotheses (i.e., of committing one or more type II errors).

The Holm–Bonferroni method also controls the FWER at ${\displaystyle \alpha }$, but with a lower increase of type II error risk than the classical Bonferroni method. The Holm–Bonferroni method sorts the p-values from lowest to highest and compares them to nominal alpha levels of ${\displaystyle {\frac {\alpha }{m}}}$ to ${\displaystyle \alpha }$ (respectively), namely the values ${\displaystyle {\frac {\alpha }{m}},{\frac {\alpha }{m-1}},\ldots ,{\frac {\alpha }{2}},{\frac {\alpha }{1}}}$.

• The index ${\displaystyle k}$ identifies the first p-value that is not low enough to validate rejection. Therefore, the null hypotheses ${\displaystyle H_{(1)},\ldots ,H_{(k-1)}}$ are rejected, while the null hypotheses ${\displaystyle H_{(k)},...,H_{(m)}}$ are not rejected.
• If ${\displaystyle k=1}$ then no p-values were low enough for rejection, therefore no null hypotheses are rejected.
• If no such index ${\displaystyle k}$ could be found then all p-values were low enough for rejection, therefore all null hypotheses are rejected (none are accepted).

Let ${\displaystyle H_{(1)}\ldots H_{(m)}}$ be the family of hypotheses sorted by their p-values ${\displaystyle P_{(1)}\leq P_{(2)}\leq \cdots \leq P_{(m)}}$. Let ${\displaystyle I_{0}}$ be the set of indices corresponding to the (unknown) true null hypotheses, having ${\displaystyle m_{0}}$ members.

Claim: If we wrongly reject some true hypothesis, there is a true hypothesis ${\displaystyle H_{(\ell )}}$ for which ${\displaystyle P_{(\ell )}}$ at most ${\displaystyle {\frac {\alpha }{m_{0}}}}$.

First note that, in this case, there is at least one true hypothesis, so ${\displaystyle m_{0}\geq 1}$. Let ${\displaystyle \ell }$ be such that ${\displaystyle H_{(\ell )}}$ is the first rejected true hypothesis. Then ${\displaystyle H_{(1)},\ldots ,H_{(\ell -1)}}$are all rejected false hypotheses. It follows that ${\displaystyle \ell -1\leq m-m_{0}}$ and, hence, ${\displaystyle {\frac {1}{m-\ell +1}}\leq {\frac {1}{m_{0}}}}$ (1). Since ${\displaystyle H_{(\ell )}}$ is rejected, it must be ${\displaystyle P_{(\ell )}\leq {\frac {\alpha }{m-\ell +1}}}$ by definition of the testing procedure. Using (1), we conclude that ${\displaystyle P_{(\ell )}\leq {\frac {\alpha }{m_{0}}}}$, as desired.

So let us define the random event ${\displaystyle A=\bigcup _{i\in I_{0}}\left\{P_{i}\leq {\frac {\alpha }{m_{0}}}\right\}}$. Note that, for ${\displaystyle i\in I_{o}}$, since ${\displaystyle H_{i}}$ is a true null hypothesis, we have that ${\displaystyle P\left(\left\{P_{i}\leq {\frac {\alpha }{m_{0}}}\right\}\right)={\frac {\alpha }{m_{0}}}}$. Subadditivity of the probability measure implies that ${\displaystyle \Pr(A)\leq \sum _{i\in I_{0}}P\left(\left\{P_{i}\leq {\frac {\alpha }{m_{0}}}\right\}\right)=\sum _{i\in I_{0}}{\frac {\alpha }{m_{0}}}=\alpha }$. Therefore, the probability to reject a true hypothesis is at most ${\displaystyle \alpha }$.

The Holm–Bonferroni method can be viewed as a closed testing procedure,[2] with the Bonferroni correction applied locally on each of the intersections of null hypotheses.

The closure principle states that a hypothesis ${\displaystyle H_{i}}$ in a family of hypotheses ${\displaystyle H_{1},\ldots ,H_{m}}$ is rejected – while controlling the FWER at level ${\displaystyle \alpha }$ – if and only if all the sub-families of the intersections with ${\displaystyle H_{i}}$ are rejected at level ${\displaystyle \alpha }$.

The Holm-Bonferroni method is a shortcut procedure, since it makes ${\displaystyle m}$ or less comparisons, while the number of all intersections of null hypotheses to be tested is of order ${\displaystyle 2^{m}}$. It controls the FWER in the strong sense.

In the Holm–Bonferroni procedure, we first test ${\displaystyle H_{(1)}}$. If it is not rejected then the intersection of all null hypotheses ${\displaystyle \bigcap \nolimits _{i=1}^{m}H_{i}}$ is not rejected too, such that there exists at least one intersection hypothesis for each of elementary hypotheses ${\displaystyle H_{1},\ldots ,H_{m}}$ that is not rejected, thus we reject none of the elementary hypotheses.

If ${\displaystyle H_{(1)}}$ is rejected at level ${\displaystyle \alpha /m}$ then all the intersection sub-families that contain it are rejected too, thus ${\displaystyle H_{(1)}}$ is rejected. This is because ${\displaystyle P_{(1)}}$ is the smallest in each one of the intersection sub-families and the size of the sub-families is at most ${\displaystyle m}$, such that the Bonferroni threshold larger than ${\displaystyle \alpha /m}$.

The same rationale applies for ${\displaystyle H_{(2)}}$. However, since ${\displaystyle H_{(1)}}$ already rejected, it sufficient to reject all the intersection sub-families of ${\displaystyle H_{(2)}}$ without ${\displaystyle H_{(1)}}$. Once ${\displaystyle P_{(2)}\leq \alpha /(m-1)}$ holds all the intersections that contains ${\displaystyle H_{(2)}}$ are rejected.

The same applies for each ${\displaystyle 1\leq i\leq m}$.

When the hypothesis tests are not negatively dependent, it is possible to replace ${\displaystyle {\frac {\alpha }{m}},{\frac {\alpha }{m-1}},\ldots ,{\frac {\alpha }{1}}}$ with:

${\displaystyle 1-(1-\alpha )^{1/m},1-(1-\alpha )^{1/(m-1)},\ldots ,1-(1-\alpha )^{1}}$

resulting in a slightly more powerful test.

Let ${\displaystyle P_{(1)},\ldots ,P_{(m)}}$ be the ordered unadjusted p-values. Let ${\displaystyle H_{(i)}}$, ${\displaystyle 0\leq w_{(i)}}$ correspond to ${\displaystyle P_{(i)}}$. Reject ${\displaystyle H_{(i)}}$ as long as

${\displaystyle P_{(j)}\leq {\frac {w_{(j)}}{\sum _{k=j}^{m}w_{(k)}}}\alpha ,\quad j=1,\ldots ,i}$

The adjusted p-values for Holm–Bonferroni method are:

${\displaystyle {\widetilde {p}}_{(i)}=\max _{j\leq i}\left\{(m-j+1)p_{(j)}\right\}_{1},{\text{ where }}\{x\}_{1}\equiv \min(x,1).}$

In the earlier example, the adjusted p-values are ${\displaystyle {\widetilde {p}}_{1}=0.03}$, ${\displaystyle {\widetilde {p}}_{2}=0.06}$, ${\displaystyle {\widetilde {p}}_{3}=0.06}$ and ${\displaystyle {\widetilde {p}}_{4}=0.02}$. Only hypotheses ${\displaystyle H_{1}}$ and ${\displaystyle H_{4}}$ are rejected at level ${\displaystyle \alpha =0.05}$.

Similar adjusted p-values for Holm-Šidák method can be defined recursively as ${\displaystyle {\widetilde {p}}_{(i)}=\max \left\{{\widetilde {p}}_{(i-1)},1-(1-p_{(i)})^{m-i+1}\right\}}$, where ${\displaystyle {\widetilde {p}}_{(1)}=1-(1-p_{(1)})^{m}}$. Due to the inequality ${\displaystyle 1-(1-\alpha )^{1/n}<\alpha /n}$ for ${\displaystyle n\geq 2}$, the Holm-Šidák method will be more powerful than Holm-Bonferroni method.

The weighted adjusted p-values are:

${\displaystyle {\widetilde {p}}_{(i)}=\max _{j\leq i}\left\{{\frac {\sum _{k=j}^{m}{w_{(k)}}}{w_{(j)}}}p_{(j)}\right\}_{1},{\text{ where }}\{x\}_{1}\equiv \min(x,1).}$

A hypothesis is rejected at level α if and only if its adjusted p-value is less than α. In the earlier example using equal weights, the adjusted p-values are 0.03, 0.06, 0.06, and 0.02. This is another way to see that using α = 0.05, only hypotheses one and four are rejected by this procedure.