Unbiased estimation of standard deviation

Correction factor ${\displaystyle c_{4}}$ versus sample size n.

When the random variable is normally distributed, a minor correction exists to eliminate the bias. To derive the correction, note that for normally distributed X, Cochran's theorem implies that ${\displaystyle (n-1)s^{2}/\sigma ^{2}}$ has a chi square distribution with ${\displaystyle n-1}$ degrees of freedom and thus its square root, ${\displaystyle {\sqrt {n-1}}s/\sigma }$ has a chi distribution with ${\displaystyle n-1}$ degrees of freedom. Consequently, calculating the expectation of this last expression and rearranging constants,

${\displaystyle \operatorname {E} [s]=c_{4}(n)\sigma }$

where the correction factor ${\displaystyle c_{4}(n)}$ is the scale mean of the chi distribution with ${\displaystyle n-1}$ degrees of freedom, ${\displaystyle \mu _{1}/{\sqrt {n-1}}}$. This depends on the sample size n, and is given as follows:[1]

${\displaystyle c_{4}(n)={\sqrt {\frac {2}{n-1}}}{\frac {\Gamma \left({\frac {n}{2}}\right)}{\Gamma \left({\frac {n-1}{2}}\right)}}=1-{\frac {1}{4n}}-{\frac {7}{32n^{2}}}-{\frac {19}{128n^{3}}}+O(n^{-4})}$

where Γ(·) is the gamma function. An unbiased estimator of σ can be obtained by dividing ${\displaystyle s}$ by ${\displaystyle c_{4}(n)}$. As ${\displaystyle n}$ grows large it approaches 1, and even for smaller values the correction is minor. The figure shows a plot of ${\displaystyle c_{4}(n)}$ versus sample size. The table below gives numerical values of ${\displaystyle c_{4}(n)}$ and algebraic expressions for some values of ${\displaystyle n}$; more complete tables may be found in most textbooks on statistical quality control.

Sample size	Expression of ${\displaystyle c_{4}}$	Numerical value
2	${\displaystyle {\sqrt {\frac {2}{\pi }}}}$	0.7978845608
3	${\displaystyle {\frac {\sqrt {\pi }}{2}}}$	0.8862269255
4	${\displaystyle 2{\sqrt {\frac {2}{3\pi }}}}$	0.9213177319
5	${\displaystyle {\frac {3}{4}}{\sqrt {\frac {\pi }{2}}}}$	0.9399856030
6	${\displaystyle {\frac {8}{3}}{\sqrt {\frac {2}{5\pi }}}}$	0.9515328619
7	${\displaystyle {\frac {5{\sqrt {3\pi }}}{16}}}$	0.9593687891
8	${\displaystyle {\frac {16}{5}}{\sqrt {\frac {2}{7\pi }}}}$	0.9650304561
9	${\displaystyle {\frac {35{\sqrt {\pi }}}{64}}}$	0.9693106998
10	${\displaystyle {\frac {128}{105}}{\sqrt {\frac {2}{\pi }}}}$	0.9726592741
100		0.9974779761
1000		0.9997497811
10000		0.9999749978
2k	${\displaystyle {\sqrt {\frac {2}{\pi (2k-1)}}}{\frac {2^{2k-2}(k-1)!^{2}}{(2k-2)!}}}$
2k+1	${\displaystyle {\sqrt {\frac {\pi }{k}}}{\frac {(2k-1)!}{2^{2k-1}(k-1)!^{2}}}}$

It is important to keep in mind this correction only produces an unbiased estimator for normally and independently distributed X. When this condition is satisfied, another result about s involving ${\displaystyle c_{4}(n)}$ is that the standard error of s is[2][3] ${\displaystyle \sigma {\sqrt {1-c_{4}^{2}}}}$, while the standard error of the unbiased estimator is ${\displaystyle \sigma {\sqrt {c_{4}^{-2}-1}}.}$

If calculation of the function c₄(n) appears too difficult, there is a simple rule of thumb[4] to take the estimator

${\displaystyle {\hat {\sigma }}={\sqrt {{\frac {1}{n-1.5}}\sum _{i=1}^{n}(x_{i}-{\overline {x}})^{2}}}}$

The formula differs from the familiar expression for s² only by having n − 1.5 instead of n − 1 in the denominator. This expression is only approximate; in fact,

${\displaystyle \operatorname {E} \left[{\hat {\sigma }}\right]=\sigma \cdot \left(1+{\frac {1}{16n^{2}}}+{\frac {3}{16n^{3}}}+O(n^{-4})\right).}$

The bias is relatively small: say, for ${\displaystyle n=3}$ it is equal to 2.3%, and for ${\displaystyle n=9}$ the bias is already 0.1%.

In cases where statistically independent data are modelled by a parametric family of distributions other than the normal distribution, the population standard deviation will, if it exists, be a function of the parameters of the model. One general approach to estimation would be maximum likelihood. Alternatively, it may be possible to use the Rao–Blackwell theorem as a route to finding a good estimate of the standard deviation. In neither case would the estimates obtained usually be unbiased. Notionally, theoretical adjustments might be obtainable to lead to unbiased estimates but, unlike those for the normal distribution, these would typically depend on the estimated parameters.

If the requirement is simply to reduce the bias of an estimated standard deviation, rather than to eliminate it entirely, then two practical approaches are available, both within the context of resampling. These are jackknifing and bootstrapping. Both can be applied either to parametrically based estimates of the standard deviation or to the sample standard deviation.

For non-normal distributions an approximate (up to O(n⁻¹) terms) formula for the unbiased estimator of the standard deviation is

${\displaystyle {\hat {\sigma }}={\sqrt {{\frac {1}{n-1.5-{\tfrac {1}{4}}\gamma _{2}}}\sum _{i=1}^{n}\left(x_{i}-{\overline {x}}\right)^{2}}},}$

where γ₂ denotes the population excess kurtosis. The excess kurtosis may be either known beforehand for certain distributions, or estimated from the data.

To illustrate the magnitude of the bias in the standard deviation, consider a dataset that consists of sequential readings from an instrument that uses a specific digital filter whose ACF is known to be given by

${\displaystyle \rho _{k}=(1-\alpha )^{k}}$

where α is the parameter of the filter, and it takes values from zero to unity. Thus the ACF is positive and geometrically decreasing.

Bias in standard deviation for autocorrelated data.

The figure shows the ratio of the estimated standard deviation to its known value (which can be calculated analytically for this digital filter), for several settings of α as a function of sample size n. Changing α alters the variance reduction ratio of the filter, which is known to be

${\displaystyle {\rm {VRR}}={\frac {\alpha }{2-\alpha }}}$

so that smaller values of α result in more variance reduction, or “smoothing.” The bias is indicated by values on the vertical axis different from unity; that is, if there were no bias, the ratio of the estimated to known standard deviation would be unity. Clearly, for modest sample sizes there can be significant bias (a factor of two, or more).

It is often of interest to estimate the variance or standard deviation of an estimated mean rather than the variance of a population. When the data are autocorrelated, this has a direct effect on the theoretical variance of the sample mean, which is[7]

${\displaystyle {\rm {Var}}\left[{\overline {x}}\right]={\frac {\sigma ^{2}}{n}}\left[1+2\sum _{k=1}^{n-1}{\left(1-{\frac {k}{n}}\right)\rho _{k}}\right].}$

The variance of the sample mean can then be estimated by substituting an estimate of σ². One such estimate can be obtained from the equation for E[s²] given above. First define the following constants, assuming, again, a known ACF:

${\displaystyle \gamma _{1}\equiv 1-{\frac {2}{n-1}}\sum _{k=1}^{n-1}{\left(1-{\frac {k}{n}}\right)}\rho _{k}}$

${\displaystyle \gamma _{2}\equiv 1+2\sum _{k=1}^{n-1}{\left(1-{\frac {k}{n}}\right)}\rho _{k}}$

so that

${\displaystyle {\rm {E}}\left[s^{2}\right]=\sigma ^{2}\gamma _{1}\Rightarrow {\rm {E}}\left[{\frac {s^{2}}{\gamma _{1}}}\right]=\sigma ^{2}}$

This says that the expected value of the quantity obtained by dividing the observed sample variance by the correction factor ${\displaystyle \gamma _{1}}$ gives an unbiased estimate of the variance. Similarly, re-writing the expression above for the variance of the mean,

${\displaystyle {\rm {Var}}\left[{\overline {x}}\right]={\frac {\sigma ^{2}}{n}}\gamma _{2}}$

and substituting the estimate for ${\displaystyle \sigma ^{2}}$ gives[8]

${\displaystyle {\rm {Var}}\left[{\overline {x}}\right]={\rm {E}}\left[{\frac {s^{2}}{\gamma _{1}}}\left({\frac {\gamma _{2}}{n}}\right)\right]={\rm {E}}\left[{\frac {s^{2}}{n}}\left\{{\frac {n-1}{{\frac {n}{\gamma _{2}}}-1}}\right\}\right]}$

which is an unbiased estimator of the variance of the mean in terms of the observed sample variance and known quantities. If the autocorrelations ${\displaystyle \rho _{k}}$ are identically zero, this expression reduces to the well-known result for the variance of the mean for independent data. The effect of the expectation operator in these expressions is that the equality holds in the mean (i.e., on average).

Having the expressions above involving the variance of the population, and of an estimate of the mean of that population, it would seem logical to simply take the square root of these expressions to obtain unbiased estimates of the respective standard deviations. However it is the case that, since expectations are integrals,

${\displaystyle {\rm {E}}[s]\neq {\sqrt {{\rm {E}}\left[s^{2}\right]}}\neq \sigma {\sqrt {\gamma _{1}}}}$

Instead, assume a function θ exists such that an unbiased estimator of the standard deviation can be written

${\displaystyle {\rm {E}}[s]=\sigma \theta {\sqrt {\gamma _{1}}}\Rightarrow {\hat {\sigma }}={\frac {s}{\theta {\sqrt {\gamma _{1}}}}}}$

and θ depends on the sample size n and the ACF. In the case of NID (normally and independently distributed) data, the radicand is unity and θ is just the c₄ function given in the first section above. As with c₄, θ approaches unity as the sample size increases (as does γ₁).

It can be demonstrated via simulation modeling that ignoring θ (that is, taking it to be unity) and using

${\displaystyle {\rm {E}}[s]\approx \sigma {\sqrt {\gamma _{1}}}\Rightarrow {\hat {\sigma }}\approx {\frac {s}{\sqrt {\gamma _{1}}}}}$

removes all but a few percent of the bias caused by autocorrelation, making this a reduced-bias estimator, rather than an unbiased estimator. In practical measurement situations, this reduction in bias can be significant, and useful, even if some relatively small bias remains. The figure above, showing an example of the bias in the standard deviation vs. sample size, is based on this approximation; the actual bias would be somewhat larger than indicated in those graphs since the transformation bias θ is not included there.

The unbiased variance of the mean in terms of the population variance and the ACF is given by

${\displaystyle {\rm {Var}}\left[{\overline {x}}\right]={\frac {\sigma ^{2}}{n}}\gamma _{2}}$

and since there are no expected values here, in this case the square root can be taken, so that

${\displaystyle \sigma _{\overline {x}}={\frac {\sigma }{\sqrt {n}}}{\sqrt {\gamma _{2}}}}$

Using the unbiased estimate expression above for σ, an estimate of the standard deviation of the mean will then be

${\displaystyle {\hat {\sigma }}_{\overline {x}}={\frac {s}{\theta {\sqrt {n}}}}{\frac {\sqrt {\gamma _{2}}}{\sqrt {\gamma _{1}}}}}$

If the data are NID, so that the ACF vanishes, this reduces to

${\displaystyle {\hat {\sigma }}_{\overline {x}}={\frac {s}{c_{4}{\sqrt {n}}}}}$

In the presence of a nonzero ACF, ignoring the function θ as before leads to the reduced-bias estimator

${\displaystyle {\hat {\sigma }}_{\overline {x}}\approx {\frac {s}{\sqrt {n}}}{\frac {\sqrt {\gamma _{2}}}{\sqrt {\gamma _{1}}}}={\frac {s}{\sqrt {n}}}{\sqrt {\frac {n-1}{{\frac {n}{\gamma _{2}}}-1}}}}$

which again can be demonstrated to remove a useful majority of the bias.