Duncan's new multiple range test

Assumptions:
1.A sample of observed means ${\displaystyle m_{1},m_{2},...,m_{n}}$ , which have been drawn independently from n normal populations with "true" means, ${\displaystyle \mu _{1},\mu _{2},...,\mu _{n}}$ respectively.
2.A common standard error ${\displaystyle \sigma }$. This standard error is unknown, but there is available the usual estimate ${\displaystyle s_{m}}$ , which is independent of the observed means and is based on a number of degrees of freedom, denoted by ${\displaystyle n_{2}}$ . (More precisely, ${\displaystyle S_{m}}$, has the property that ${\displaystyle {\frac {n_{2}\cdot S_{m}^{2}}{\sigma _{m}^{2}}}}$ is distributed as ${\displaystyle \chi ^{2}}$ with ${\displaystyle n_{2}}$ degrees of freedom, independently of sample means).

The exact definition of the test is:

The difference between any two means in a set of n means is significant provided the range of each and every subset which contains the given means is significant according to an ${\displaystyle \alpha _{p}}$ level range test where ${\displaystyle \alpha _{p}=1-\gamma _{p}}$ , ${\displaystyle \gamma _{p}=(1-\alpha )^{(p-1)}}$ and ${\displaystyle p}$ is the number of means in the subset concerned.

Exception: The sole exception to this rule is that no difference between two means can be declared significant if the two means concerned are both contained in a subset of the means which has a non-significant range.

       1.Rank the sample means, largest to smallest.
       2. For each ${\displaystyle m_{i}}$ sample mean, largest to smallest, do the following:
       2.1 for each sample mean, (denoted ${\displaystyle m_{j}}$), for smallest up to ${\displaystyle m_{(i-1)}}$.
       2.1.1 compare ${\displaystyle m_{i}-m_{j}}$ to critical value ${\displaystyle \sigma _{m}\cdot R_{(p,\alpha )}}$,${\displaystyle P=i-j,\alpha =\alpha _{p}}$
       2.1.2 if ${\displaystyle m_{i}-m_{j}}$ does not exceed the critical value, the subset ${\displaystyle (m_{j},m_{j+1},...,m_{I})}$ is declared not significantly different:
               2.1.2.1 Go to next iteration of loop 2.
       2.1.3 Otherwise, keep going with loop 2.1

Let us look at the example of 5 treatment means:

With a standard error of ${\displaystyle s_{m}=1.796}$, and ${\displaystyle \nu =20}$ (degrees of freedom for estimating the standard error). Using a known tabulation for Q, one reaches the values of ${\displaystyle r_{(p,\nu ,\alpha )}}$:

${\displaystyle r_{(2,20,0.05)}=2.95}$
${\displaystyle r_{(3,20,0.05)}=3.10}$
${\displaystyle r_{(4,20,0.05)}=3.18}$
${\displaystyle r_{(5,20,0.05)}=3.25}$

Now we may obtain the values of the shortest significant range, by the formula:
${\displaystyle R_{(p,\nu ,\alpha )}=\sigma _{m}*r_{(p,\nu ,\alpha )}}$

Reaching:

${\displaystyle R_{(2,20,0.05)}=3.75}$
${\displaystyle R_{(3,20,0.05)}=3.94}$
${\displaystyle R_{(4,20,0.05)}=4.04}$
${\displaystyle R_{(5,20,0.05)}=4.13}$

Then, the observed differences between means are tested, beginning with the largest versus smallest, which would be compared with the least significant range ${\displaystyle R_{(5,20,0.05)}=4.13.}$ Next, the difference of the largest and the second smallest is computed and compared with the least significant difference ${\displaystyle R_{(4,20,0.05)}=4.04}$.

If an observed difference is greater than the corresponding shortest significant range, then we conclude that the pair of means in question is significantly different. If an observed difference is smaller than the corresponding shortest significant range, all differences sharing the same upper mean are considered insignificant, in order to prevent contradictions (differences sharing the same upper mean are shorter by construction).
 
For our case, the comparison will yield:

${\displaystyle 4vs.1:21.6-9.8=11.8>4.13(R_{5})}$
${\displaystyle 4vs.5:21.6-10.8=10.8>4.04(R_{4})}$
${\displaystyle 4vs.2:21.6-15.4=6.2>3.94(R_{3})}$
${\displaystyle 4vs.3:21.6-17.6=4.0>3.75(R_{2})}$
${\displaystyle 3vs.1:17.6-9.8=7.8>4.04(R_{4})}$
${\displaystyle 3vs.5:17.6-10.8=6.8>3.94(R_{3})}$
${\displaystyle 3vs.2:17.6-15.4=2.2<3.75(R_{2})}$
${\displaystyle 2vs.1:15.4-9.8=5.6>3.94(R_{3})}$
${\displaystyle 2vs.5:15.4-10.8=4.6>3.75(R_{2})}$
${\displaystyle 5vs.1:10.8-9.8=1.0<3.75(R_{2})}$

We see that there are significant differences between all pairs of treatments except (T3,T2) and (T5,T1). A graph underlining those means that are not significantly different is shown below:
T1 T5 T2 T3 T4

The new multiple range test proposed by Duncan makes use of special protection levels based upon degrees of freedom. Let ${\displaystyle \gamma _{2,\alpha }={1-\alpha }}$ be the protection level for testing the significance of a difference between two means; that is, the probability that a significant difference between two means will not be found if the population means are equal. Duncan reasons that one has p-1 degrees of freedom for testing p ranked mean, and hence one may conduct p-1 independent tests, each with protection level ${\displaystyle \gamma _{2,\alpha }={1-\alpha }}$ . Hence, the joint protection level is:

${\displaystyle \gamma _{p,\alpha }=\gamma _{2,\alpha }^{p-1}=(1-\alpha )^{p-1}}$ where ${\displaystyle \alpha _{p}=1-\gamma _{p}}$

that is, the probability that one finds no significant differences in making p-1 independent tests, each at protection level ${\displaystyle \gamma _{2,\alpha }={1-\alpha }}$, is ${\displaystyle \gamma _{2,\alpha }^{p-1}}$, under the hypothesis that all p population means are equal. In general: the difference between any two means in a set of n means is significant provided the range of each and every subset, which contains the given means, is significant according to an ${\displaystyle \alpha _{p}}$ –level range test, where p is the number of means in the subset concerned.

For ${\displaystyle \alpha =0.05}$ , the protection level can be tabulated for various value of r as follows:

	Protection level ${\displaystyle$ :\gamma _{p,\alpha }}	probability of falsely rejecting ${\displaystyle H_{0}:\alpha _{p}}$
p=2	0.95	0.05
p=3	0.903	0.097
p=4	0.857	0.143
p=5	0.815	0.185
p=6	0.774	0.226
p=7	0.735	0.265

Note that although this procedure makes use of the Studentized range, his error rate is neither on an experiment-wise basis (as with Tukey's) nor on a per- comparisons basis. Duncan's multiple range test does not control the family-wise error rate. See Criticism Section for further details.

Duncan (1965) also gave the first Bayesian multiple comparison procedure, for the pairwise comparisons among the means in a one-way layout. This multiple comparison procedure is different for the one discussed above.

Duncan's Bayesian MCP discusses the differences between ordered group means, where the statistics in question are pairwise comparison (no equivalent is defined for the property of a subset having 'significantly different' property).

Duncan modeled the consequences of two or more means being equal using additive loss functions within and across the pairwise comparisons. If one assumes the same loss function across the pairwise comparisons, one needs to specify only one constant K, and this indicates the relative seriousness of type I to type II errors in each pairwise comparison.

A study, which performed by Juliet Popper Shaffer (1998), has shown that the method proposed by Duncan, modified to provide weak control of FWE and using an empirical estimate of the variance of the population means, has good properties both from the Bayesian point of view, as a minimum- risk method, and from the frequentist point of view, with good average power.

In addition, results indicate considerable similarity in both risk and average power between Duncan's modified procedure and the Benjamini and Hochberg (1995) False discovery rate -controlling procedure, with the same weak family-wise error control.

Duncan's test has been criticised as being too liberal by many statisticians including Henry Scheffé, and John W. Tukey. Duncan argued that a more liberal procedure was appropriate because in real world practice the global null hypothesis H₀ = "All means are equal" is often false and thus traditional statisticians overprotect a probably false null hypothesis against type I errors. According to Duncan, one should adjust the protection levels for different p-mean comparisons according to the problem discussed. The example discussed by Duncan in his 1955 paper is of a comparison of many means (i.e. 100),when one is interested only in two-mean and three-mean comparisons, and general p-mean comparisons (deciding whether there is some difference between p-means) are of no special interest (if p is 15 or more for example). Duncan's multiple range test is very “liberal” in terms of Type I errors. The following example will illustrate why:

Let us assume one is truly interested, as Duncan suggested, only with the correct ranking of subsets of size 4 or below. Let us also assume that one performs the simple pairwise comparison with a protection level ${\displaystyle \gamma _{2}=0.95}$. Given an overall set of 100 means, let us look at the null hypotheses of the test:

There are ${\displaystyle 100 \choose 2}$ null hypotheses for the correct ranking of each 2 means. The significance level of each hypothesis is ${\displaystyle 1-0.95=0.05}$

There are ${\displaystyle 100 \choose 3}$ null hypotheses for the correct ranking of each 3 means. The significance level of each hypothesis is ${\displaystyle 1-(0.95)^{2}=0.097}$

There are ${\displaystyle 100 \choose 4}$ null hypotheses for the correct ranking of each 4 means. The significance level of each hypothesis is ${\displaystyle 1-(0.95)^{3}=0.143}$

As we can see, the test has two main problems, regarding the type I errors:

1. Duncan’s tests is based on the Newman–Keuls procedure, which does not protect the family-wise error rate (though protecting the per-comparison alpha level)
2. Duncan’s test intentionally raises the alpha levels (Type I error rate) in each step of the Newman–Keuls procedure (significance levels of ${\displaystyle \alpha _{p}\geq \alpha }$).

Therefore, it is advised not to use the procedure discussed.

Duncan later developed the Duncan–Waller test which is based on Bayesian principles. It uses the obtained value of F to estimate the prior probability of the null hypothesis being true.

• Duncan, D. B. (1955). "Multiple range and multiple F tests". Biometrics. 11 (1): 1–42. doi:10.2307/3001478. JSTOR 3001478.
• Shaffer, Juliet Popper (1999). "A semi-Bayesian study of Duncan's Bayesian multiple comparison procedure". Journal of Statistical Planning and Inference. 82 (1–2): 197–213. doi:10.1016/S0378-3758(99)00042-7.
• Berry, Donald A.; Hochberg, Yosef (1999). "Bayesian perspectives on multiple comparisons". Journal of Statistical Planning and Inference. 82 (1–2): 215–227. doi:10.1016/S0378-3758(99)00044-0.
• Parsad, Rajender. "Multiple comparison Procedures" (Document). I.A.S.R.I, Library Avenue, New Delhi 110012.

Tables for the Use of Range and Studentized Range in Tests of Hypotheses

• H. Leon Harter, Champaigne, IL; N. Balakrishnan, McMaster University, Hamilton, Ontario, Canada; Hardback - Published Oct 27, 1997

• Critical values for Duncan's multiple range tests