Scott's Pi

Scott's pi is similar to Cohen's kappa in that they improve on simple observed agreement by factoring in the extent of agreement that might be expected by chance. However, in each statistic, the expected agreement is calculated slightly differently. Scott's pi makes the assumption that annotators have the same distribution of responses, which makes Cohen's kappa slightly more informative. Scott's pi is extended to more than two annotators by Fleiss' kappa.

The equation for Scott's pi, as in Cohen's kappa, is:

${\displaystyle \pi ={\frac {\Pr(a)-\Pr(e)}{1-\Pr(e)}},}$

However, Pr(e) is calculated using squared "joint proportions" which are squared arithmetic means of the marginal proportions (whereas Cohen's uses squared geometric means of them).

Confusion matrix for two annotators, three categories {Yes, No, Maybe} and 45 items rated (90 ratings for 2 annotators):

To calculate the expected agreement, sum marginals across annotators and divide by the total number of ratings to obtain joint proportions. Square and total these:

To calculate observed agreement, divide the number of items on which annotators agreed by the total number of items. In this case,

${\displaystyle \Pr(a)={\frac {1+5+9}{45}}=0.333.}$

Given that Pr(e) = 0.369, Scott's pi is then

${\displaystyle \pi ={\frac {0.333-0.369}{1-0.369}}=-0.057.}$

• Krippendorff's alpha

• Scott, W. (1955). "Reliability of content analysis: The case of nominal scale coding." Public Opinion Quarterly, 19(3), 321–325.
• Krippendorff, K. (2004b) “Reliability in content analysis: Some common misconceptions and recommendations.” in Human Communication Research. Vol. 30, pp. 411–433.