Good–Turing frequency estimation

• Assuming that ${\displaystyle X}$ distinct species have been observed, enumerated ${\displaystyle \;1,\dots ,X~.}$
• Then the frequency vector, ${\displaystyle \,{\bar {R}}\;,}$ has elements ${\displaystyle \,R_{x}\,}$ that give the number of individuals that have been observed for species ${\displaystyle \,x\;.}$
• The frequency of frequencies vector, ${\displaystyle \,(N_{r})_{r=0,1,\ldots }\;,}$ shows how many times the frequency r occurs in the vector ${\displaystyle {\bar {R}}}$ (i.e. among the elements ${\displaystyle R_{x}}$):

${\displaystyle N_{r}={\Bigl |}\left\{x\mid R_{x}=r\right\}{\Bigr |}~.}$

For example, ${\displaystyle N_{1}}$ is the number of species for which only one individual was observed. Note that the total number of objects observed, ${\displaystyle \,N\,,}$ can be found from

${\displaystyle N=\sum _{r=1}^{\infty }rN_{r}=\sum _{x=1}^{X}R_{x}~.}$

The first step in the calculation is to estimate the probability that a future observed individual (or the next observed individual) is a member of a thus far unseen species. This estimate is:[8]

${\displaystyle p_{0}={\frac {N_{1}}{N}}~.}$

The next step is to estimate the probability that the next observed individual is from a species which has been seen ${\displaystyle r}$ times. For a single species this estimate is:

${\displaystyle p_{r}={\frac {\;(r+1)\;S(N_{r+1})\;}{N\,S(N_{r})}}~.}$

Here, the notation ${\displaystyle S(\cdot )}$ means the smoothed, or adjusted value of the frequency shown in parentheses. An overview of how to perform this smoothing follows in the next section (see also empirical Bayes method).

To estimate the probability that the next observed individual is from any species from this group (i.e., the group of species seen ${\displaystyle \,r\,}$ times) one can use the following formula:

${\displaystyle {\frac {\;(r+1)\,S(N_{r+1})\;}{N}}~.}$

For smoothing the erratic values in ${\displaystyle \,N_{r}\,}$ for large r, we would like to make a plot of ${\displaystyle \;\log N_{r}\;}$ versus ${\displaystyle \;\log r\;}$ but this is problematic because for large ${\displaystyle \,r\,}$ many ${\displaystyle \,N_{r}\,}$ will be zero. Instead a revised quantity, ${\displaystyle \;\log Z_{r}\;,}$ is plotted versus ${\displaystyle \;\log r~,}$ where ${\displaystyle \,Z_{r}\,}$ is defined as

${\displaystyle Z_{r}={\frac {N_{r}}{\;{\tfrac {1}{2}}\,(t-q)\;}}~,}$

and where q, r, and t are three consecutive subscripts with non-zero counts ${\displaystyle \;N_{q},N_{r},N_{t}\;}$. For the special case when r is 1, take q to be 0. In the opposite special case, when ${\displaystyle \,r=r_{\mathsf {last}}\;}$ is the index of the last non-zero count, replace the divisor ${\displaystyle \,{\tfrac {1}{2}}\,(t-q)\,}$ with ${\displaystyle \,r_{\mathsf {last}}-q\;,}$ so ${\displaystyle \;Z_{r_{\mathsf {last}}}={\frac {N_{r_{\mathsf {last}}}}{\;r_{\mathsf {last}}-q\;}}~.}$

A simple linear regression is then fitted to the log–log plot.

For small values of ${\displaystyle \,r\,}$ it is reasonable to set ${\displaystyle \;S(N_{r})=N_{r}\;}$ – that is, no smoothing is performed.

For large values of r, values of ${\displaystyle \;S(N_{r})\;}$ are read off the regression line. An automatic procedure (not described here) can be used to specify at what point the switch from no smoothing to linear smoothing should take place.[9] Code for the method is available in the public domain.[10]