Binomial proportion confidence interval

Plotting the normal approximation interval on an arbitrary logistic curve reveals problems of overshoot and zero-width intervals.[2]

A commonly used formula for a binomial confidence interval relies on approximating the distribution of error about a binomially-distributed observation, ${\displaystyle {\hat {p}}}$, with a normal distribution.[3] This approximation is based on the central limit theorem and is unreliable when the sample size is small or the success probability is close to 0 or 1.[4]

Using the normal approximation, the success probability p is estimated as

${\displaystyle {\hat {p}}\pm z{\sqrt {\frac {{\hat {p}}\left(1-{\hat {p}}\right)}{n}}},}$

or the equivalent

${\displaystyle {\frac {n_{S}}{n}}\pm {\frac {z}{n{\sqrt {n}}}}{\sqrt {n_{S}n_{F}}},}$

where ${\displaystyle {\hat {p}}=n_{S}/n}$ is the proportion of successes in a Bernoulli trial process, measured with ${\displaystyle n}$ trials yielding ${\displaystyle n_{S}}$ successes and ${\displaystyle n_{F}=n-n_{S}}$ failures, and ${\displaystyle z}$ is the ${\displaystyle 1-{\tfrac {\alpha }{2}}}$ quantile of a standard normal distribution (i.e., the probit) corresponding to the target error rate ${\displaystyle \alpha }$. For a 95% confidence level, the error ${\displaystyle \alpha =1-0.95=0.05}$, so ${\displaystyle 1-{\tfrac {\alpha }{2}}=0.975}$ and ${\displaystyle z=1.96}$. From this one finds two problems. First, for ${\displaystyle {\hat {p}}}$ approaching unit (or zero), the interval narrows to zero width (implying certainty). Second, for values of ${\displaystyle {\hat {p}}<{\tfrac {z^{2}}{z^{2}+n}}}$ (or equivalently for ${\displaystyle 1-{\hat {p}}}$), the interval boundaries exceed ${\displaystyle [0,1]}$ (overshoot).

An important theoretical derivation of this confidence interval involves the inversion of a hypothesis test. Under this formulation, the confidence interval represents those values of the population parameter that would have large p-values if they were tested as a hypothesized population proportion. The collection of values, ${\displaystyle \theta }$, for which the normal approximation is valid can be represented as

${\displaystyle \left\{\theta \,\,{\bigg |}\,\,y\leq {\frac {{\hat {p}}-\theta }{\sqrt {{\frac {1}{n}}{\hat {p}}\left(1-{\hat {p}}\right)}}}\leq z_{\alpha /2}\right\},}$

where ${\displaystyle y}$ is the ${\displaystyle {\tfrac {\alpha }{2}}}$ quantile of a standard normal distribution.

Since the test in the middle of the inequality is a Wald test, the normal approximation interval is sometimes called the Wald interval or Wald method, after Abraham Wald, but it was first described by Pierre-Simon Laplace in 1812.[5]

Wilson score intervals plotted on a logistic curve, revealing asymmetry and good performance for small n and where p is at or near 0 or 1.

The Wilson score interval is an improvement over the normal approximation interval in multiple respects. It was developed by Edwin Bidwell Wilson (1927).[8] Unlike the symmetric normal approximation interval (above), the Wilson score interval is asymmetric. It does not suffer from problems of overshoot and zero-width intervals that afflict the normal interval, and it may be safely employed with small samples and skewed observations.[3] The observed coverage probability is consistently closer to the nominal value, ${\displaystyle 1-\alpha }$.[2]

Like the normal interval, the interval can be computed directly from a formula.

Wilson started with the normal approximation to the binomial:

${\displaystyle z\approx {\frac {~\left(\,p-{\hat {p}}\,\right)~}{\sigma _{n}}}}$

with the analytic formula for the sample standard deviation given by

$${\displaystyle \sigma _{n}={\sqrt {\,{\frac {\,p\left(1-p\right)\,}{n}}~}}~.}$$

Combining the two, and squaring out the radical, gives an equation that is quadratic in p:

${\displaystyle \left(\,{\hat {p}}-p\,\right)^{2}=z^{2}\cdot {\frac {\,p\left(1-p\right)\,}{n}}}$

Transforming the relation into a standard-form quadratic equation for p, treating ${\displaystyle {\hat {p}}}$ and n as known values from the sample (see prior section), and using the value of z that corresponds to the desired confidence for the estimate of p gives this:

$${\displaystyle \left(1+{\frac {\,z^{2}\,}{n}}\right)p^{2}+\left(-2{\hat {p}}-{\frac {\,z^{2}\,}{n}}\right)p+{\biggl (}{\hat {p}}^{2}{\biggr )}=0~,}$$

where all of the values in parentheses are known quantities. The solution for p estimates the upper and lower limits of the confidence interval for p. Hence the probability of success p is estimated by

${\displaystyle p\approx (w^{-},w^{+})={\frac {1}{~1+{\frac {\,z^{2}\,}{n}}~}}\left({\hat {p}}+{\frac {\,z^{2}\,}{2n}}\right)~\pm ~{\frac {z}{~1+{\frac {z^{2}}{n}}~}}{\sqrt {{\frac {\,{\hat {p}}(1-{\hat {p}})\,}{n}}+{\frac {\,z^{2}\,}{4n^{2}}}~}}~}$

or the equivalent

${\displaystyle p\approx {\frac {~n_{S}+{\tfrac {1}{2}}z^{2}~}{n+z^{2}}}~\pm ~{\frac {z}{n+z^{2}}}{\sqrt {{\frac {~n_{S}\,n_{F}~}{n}}+{\frac {z^{2}}{4}}~}}~.}$

The practical observation from using this interval is that it has good properties even for a small number of trials and / or an extreme probability.

Intuitively, the center value of this interval is the weighted average of ${\displaystyle {\hat {p}}}$ and ${\displaystyle {\tfrac {1}{2}}}$, with ${\displaystyle {\hat {p}}}$ receiving greater weight as the sample size increases. Formally, the center value corresponds to using a pseudocount of 1/2 z², the number of standard deviations of the confidence interval: add this number to both the count of successes and of failures to yield the estimate of the ratio. For the common two standard deviations in each direction interval (approximately 95% coverage, which itself is approximately 1.96 standard deviations), this yields the estimate ${\displaystyle (n_{S}+2)/(n+4)}$, which is known as the "plus four rule".

Although the quadratic can be solved explicitly, in most cases Wilson's equations can also be solved numerically using the fixed-point iteration

${\displaystyle p_{k+1}={\hat {p}}\pm z\cdot {\sqrt {\frac {p_{k}\cdot \left(1-p_{k}\right)}{n}}}}$

with ${\displaystyle p_{0}={\hat {p}}}$.

The Wilson interval can also be derived from the single sample z-test or Pearson's chi-squared test with two categories. The resulting interval,

${\displaystyle \left\{\theta \,\,{\bigg |}\,\,y\leq {\frac {{\hat {p}}-\theta }{\sqrt {{\tfrac {1}{n}}\theta (1-\theta )}}}\leq z\right\},}$

can then be solved for ${\displaystyle \theta }$ to produce the Wilson score interval. The test in the middle of the inequality is a score test.

The interval equality principle

The probability density function for the Wilson score interval, plus pdfs at interval bounds. Tail areas are equal.

Since the interval is derived by solving from the normal approximation to the binomial, the Wilson score interval ${\displaystyle (w^{-},w^{+})}$ has the property of being guaranteed to obtain the same result as the equivalent z-test or chi-squared test.

This property can be visualised by plotting the probability density function for the Wilson score interval (see Wallis 2021: 297-313)[9] and then plotting a normal pdf at each bound. The tail areas of the resulting Wilson and normal distributions, representing the chance of a significant result in that direction, must be equal.

The continuity-corrected Wilson score interval and the Clopper-Pearson interval are also compliant with this property. The practical import is that these intervals may be employed as significance tests, with identical results to the source test, and new tests may be derived by geometry.[9]

The Jeffreys interval has a Bayesian derivation, but it has good frequentist properties. In particular, it has coverage properties that are similar to those of the Wilson interval, but it is one of the few intervals with the advantage of being equal-tailed (e.g., for a 95% confidence interval, the probabilities of the interval lying above or below the true value are both close to 2.5%). In contrast, the Wilson interval has a systematic bias such that it is centred too close to p = 0.5.[10]

The Jeffreys interval is the Bayesian credible interval obtained when using the non-informative Jeffreys prior for the binomial proportion p. The Jeffreys prior for this problem is a Beta distribution with parameters (1/2, 1/2), it is a conjugate prior. After observing x successes in n trials, the posterior distribution for p is a Beta distribution with parameters (x + 1/2, n – x + 1/2).

When x ≠0 and x ≠ n, the Jeffreys interval is taken to be the 100(1 – α)% equal-tailed posterior probability interval, i.e., the α / 2 and 1 – α / 2 quantiles of a Beta distribution with parameters (x + 1/2, n – x + 1/2). These quantiles need to be computed numerically, although this is reasonably simple with modern statistical software.

In order to avoid the coverage probability tending to zero when p → 0 or 1, when x = 0 the upper limit is calculated as before but the lower limit is set to 0, and when x = n the lower limit is calculated as before but the upper limit is set to 1.[4]

The Clopper–Pearson interval is an early and very common method for calculating binomial confidence intervals.[11] This is often called an 'exact' method, as is attains the nominal coverage level in an exact sense, meaning that the coverage level never is less than the nominal ${\displaystyle 1-\alpha }$.[2]

The Clopper–Pearson interval can be written as

${\displaystyle S_{\leq }\cap S_{\geq }}$

or equivalently,

${\displaystyle \left(\inf S_{\geq }\,,\,\sup S_{\leq }\right)}$

with

${\displaystyle S_{\leq }:=\left\{p\,\,{\Big |}\,\,P\left[\operatorname {Bin} \left(n;p\right)\leq x\right]>{\frac {\alpha }{2}}\right\}{\text{ and }}S_{\geq }:=\left\{p\,\,{\Big |}\,\,P\left[\operatorname {Bin} \left(n;p\right)\geq x\right]>{\frac {\alpha }{2}}\right\},}$

where 0 ≤ x ≤ n is the number of successes observed in the sample and Bin(n;  p) is a binomial random variable with n trials and probability of success  p.

Equivalently we can say that the Clopper–Pearson interval is ${\textstyle \left({\frac {x}{n}}-\varepsilon _{1},\ {\frac {x}{n}}+\varepsilon _{2}\right)}$ with confidence level ${\displaystyle 1-\alpha }$ if ${\displaystyle \varepsilon _{i}}$ is the infimum of those such that the following tests of hypothesis succeed with significance ${\textstyle {\frac {\alpha }{2}}}$:

1. H₀: ${\displaystyle p={\frac {x}{n}}-\varepsilon _{1}}$ with H_A: ${\displaystyle p>{\frac {x}{n}}-\varepsilon _{1}}$
2. H₀: ${\displaystyle p={\frac {x}{n}}+\varepsilon _{2}}$ with H_A: ${\displaystyle p<{\frac {x}{n}}+\varepsilon _{2}}$.

Because of a relationship between the binomial distribution and the beta distribution, the Clopper–Pearson interval is sometimes presented in an alternate format that uses quantiles from the beta distribution.[12]

${\displaystyle B\left({\frac {\alpha }{2}};x,n-x+1\right)<p<B\left(1-{\frac {\alpha }{2}};x+1,n-x\right)}$

where x is the number of successes, n is the number of trials, and B(p; v,w) is the pth quantile from a beta distribution with shape parameters v and w.

Thus, ${\displaystyle p_{\min }<p<p_{\max }}$, where:

${\displaystyle {\frac {\Gamma (n+1)}{\Gamma (x)\Gamma (n-x+1)}}\int _{0}^{p_{\min }}t^{x-1}(1-t)^{n-x}dt={\frac {\alpha }{2}}}$

${\displaystyle {\frac {\Gamma (n+1)}{\Gamma (x+1)\Gamma (n-x)}}\int _{0}^{p_{\max }}t^{x}(1-t)^{n-x-1}dt=1-{\frac {\alpha }{2}}}$

The binomial proportion confidence interval is then ${\displaystyle (p_{\min },p_{\max })}$, as follows from the relation between the Binomial distribution cumulative distribution function and the regularized incomplete beta function.

When ${\displaystyle x}$ is either ${\displaystyle 0}$ or ${\displaystyle n}$, closed-form expressions for the interval bounds are available: when ${\displaystyle x=0}$ the interval is ${\textstyle \left(0,\,1-\left({\frac {\alpha }{2}}\right)^{\frac {1}{n}}\right)}$ and when ${\displaystyle x=n}$ it is ${\textstyle \left(\left({\frac {\alpha }{2}}\right)^{\frac {1}{n}},\,1\right)}$.[12]

The beta distribution is, in turn, related to the F-distribution so a third formulation of the Clopper–Pearson interval can be written using F quantiles:

${\displaystyle \left(1+{\frac {n-x+1}{(x)\,F\!\left[{\frac {\alpha }{2}};2(x),2(n-x+1)\right]}}\right)^{-1}<p<\left(1+{\frac {n-x}{(x+1)\,\,F\!\left[1-{\frac {\alpha }{2}};2(x+1),2(n-x)\right]}}\right)^{-1}}$

where x is the number of successes, n is the number of trials, and F(c; d₁, d₂) is the c quantile from an F-distribution with d₁ and d₂ degrees of freedom.[13]

The Clopper–Pearson interval is an exact interval since it is based directly on the binomial distribution rather than any approximation to the binomial distribution. This interval never has less than the nominal coverage for any population proportion, but that means that it is usually conservative. For example, the true coverage rate of a 95% Clopper–Pearson interval may be well above 95%, depending on n and  p.[4] Thus the interval may be wider than it needs to be to achieve 95% confidence, and wider than other intervals. In contrast, it is worth noting that other confidence interval may have coverage levels that are lower than the nominal ${\displaystyle 1-\alpha }$, i.e., the normal approximation (or "standard") interval, Wilson interval,[8] Agresti–Coull interval,[13] etc., with a nominal coverage of 95% may in fact cover less than 95%,[4] even for large sample sizes.[12]

The definition of the Clopper–Pearson interval can also be modified to obtain exact confidence intervals for different distributions. For instance, it can also be applied to the case where the samples are drawn without replacement from a population of a known size, instead of repeated draws of a binomial distribution. In this case, the underlying distribution would be the hypergeometric distribution.

The interval boundaries are easily computed with numerical methods functions like qbeta in R and scipy.stats.beta.ppf in Python.

from scipy.stats import beta
k = 20
n = 400
alpha = 0.05
p_u, p_o = beta.ppf([alpha/2, 1 - alpha/2], [k, k + 1], [n - k + 1, n - k])

The Agresti–Coull interval is also another approximate binomial confidence interval.[13]

Given ${\displaystyle n_{S}}$ successes in ${\displaystyle n}$ trials, define

${\displaystyle {\tilde {n}}=n+z^{2}}$

and

${\displaystyle {\tilde {p}}={\frac {1}{\tilde {n}}}\left(n_{S}+{\frac {z^{2}}{2}}\right)}$

Then, a confidence interval for ${\displaystyle p}$ is given by

${\displaystyle {\tilde {p}}\pm z{\sqrt {{\frac {\tilde {p}}{\tilde {n}}}\left(1-{\tilde {p}}\right)}}}$

where ${\displaystyle z=\Phi ^{-1}\!\left(1-{\frac {\alpha }{2}}\!\right)}$ is the quantile of a standard normal distribution, as before (for example, a 95% confidence interval requires ${\displaystyle \alpha =0.05}$, thereby producing ${\displaystyle z=1.96}$). According to Brown, Cai, and DasGupta,[4] taking ${\displaystyle z=2}$ instead of 1.96 produces the "add 2 successes and 2 failures" interval previously described by Agresti and Coull.[13]

This interval can be summarised as employing the centre-point adjustment, ${\displaystyle {\tilde {p}}}$, of the Wilson score interval, and then applying the Normal approximation to this point.[3][4]

${\displaystyle {\tilde {p}}={\frac {{\hat {p}}+{\frac {z^{2}}{2n}}}{1+{\frac {z^{2}}{n}}}}}$

The arcsine transformation has the effect of pulling out the ends of the distribution.[14] While it can stabilize the variance (and thus confidence intervals) of proportion data, its use has been criticized in several contexts.[15]

Let X be the number of successes in n trials and let p = X/n. The variance of p is

${\displaystyle \operatorname {var} (p)={\frac {p(1-p)}{n}}.}$

Using the arc sine transform the variance of the arcsine of p^1/2 is[16]

${\displaystyle \operatorname {var} \left(\arcsin {\sqrt {p}}\right)\approx {\frac {\operatorname {var} (p)}{4p(1-p)}}={\frac {p(1-p)}{4np(1-p)}}={\frac {1}{4n}}.}$

So, the confidence interval itself has the following form:

${\displaystyle \left(\sin \left(\arcsin {\sqrt {p}}-{\frac {z}{2{\sqrt {n}}}}\right)\right)^{2}<\theta <\left(\sin \left(\arcsin {\sqrt {p}}+{\frac {z}{2{\sqrt {n}}}}\right)\right)^{2}}$

where ${\displaystyle z}$ is the ${\displaystyle \scriptstyle 1\,-\,{\frac {\alpha }{2}}}$ quantile of a standard normal distribution.

This method may be used to estimate the variance of p but its use is problematic when p is close to 0 or 1.

Let p be the proportion of successes. For 0 ≤ a ≤ 2,

${\displaystyle t_{a}=\log \left({\frac {p^{a}}{(1-p)^{2-a}}}\right)=a\log(p)-(2-a)\log(1-p)}$

This family is a generalisation of the logit transform which is a special case with a = 1 and can be used to transform a proportional data distribution to an approximately normal distribution. The parameter a has to be estimated for the data set.

The rule of three is used to provide a simple way of stating an approximate 95% confidence interval for p, in the special case that no successes (${\displaystyle {\hat {p}}=0}$) have been observed.[17] The interval is (0,3/n).

By symmetry, in the case of only successes (${\displaystyle {\hat {p}}=1}$), the interval is (1 − 3/n,1).

There are several research papers that compare these and other confidence intervals for the binomial proportion.[3][2][18][19] Both Agresti and Coull (1998)[13] and Ross (2003)[20] point out that exact methods such as the Clopper–Pearson interval may not work as well as certain approximations. The Normal approximation interval and its presentation in textbooks has been heavily criticised, with many statisticians advocating that it be not used.[4] The principal problems are overshoot (bounds exceed [0, 1]), zero-width intervals at ${\displaystyle {\hat {p}}}$ = 0 and 1 (falsely implying certainty),[2] and overall inconsistency with significance testing.[3]

Of the approximations listed above, Wilson score interval methods (with or without continuity correction) have been shown to be the most accurate and the most robust,[3][4][2] though some prefer the Agresti–Coull approach for larger sample sizes.[4] Wilson and Clopper–Pearson methods obtain consistent results with source significance tests,[9] and this property is decisive for many researchers.

Many of these intervals can be calculated in R using packages like "binom".

• Binomial distribution#Confidence intervals
• Estimation theory
• Pseudocount