01111. Crosstabs: Kappa

Kappa is a measure of agreement. It is currently gaining popularity as a measure of scorer reliability. The following example is based on Cohen (1960). Suppose that you ask 200 sets of fathers and mothers to identify which of three personality descriptions best describes their oldest child. The data are saved in the file kappa.sav. The variables in kappa.sav are given in Table 1.

This data file makes use of a weight variable, called count, to specify the number of cases for each cell in the 3 (fathers' descriptions) x 3 (mothers' descriptions) contingency table. The count variable specifies the number of cases in each cell of the 3 x 3 table. The values in kappa.sav are shown in Table 2.

The count value of 88 for the first row indicates that there are 88 cases in cell 11 (fathers 1, mothers1). That is, the fathers and mothers agree that personality description #1 is the best description for their oldest child. There are 10 cases in cell 12 (fathers1, mothers 2); 2 cases in cell 13 (fathers1, mothers3); and 14 cases in cell 21 (fathers 2, mothrs 1), and so forth. By adding all the values for count you find the total number of cases, N = 200.

Please note that normally you would enter the data for each case. If the weight variable were not used, then the dataset would need to have 200 cases. The kappa.sav file uses only 9 cases to represent the entire set of data points. The weight variable is useful if you wish to reanalyze a published set of data. As in this example, you could enter the cell frequences as a weight variable rather than entering all the individual cases.

The kappa statistic is found in crosstabs. But there is an additional step that needs to be taken before you enter the crosstabs dialog menu, you need to tell SPSS that the variable count is to be used as a weighting variable.

Select

Data
    Weight Cases

In the dialog box select the Weight cases by radio box and then move the count variable to the Frequency Variable: window. The press OK to run the command.

Kappa is in the statistics option box in the crosstabs procedure. Click on

Statistics
      Summarize
            Crosstabs
                 Statistics
                      Kappa

In this discussion of kappa the variable fathers is designated as the row variable and the variable mothers is designated as the column variable. The cell options selected are observed counts, expected counts, and row (father) percents. All statisitics other than kappa have been suppressed.

The frequencies table for the kappa data is shown in Table 3 and the kappa statistic output is shown in Table 4.

The proportion of agreements after chance has been excluded is 49%, kappa (N = 200) = .492, p < .0005.

One way to compute an index of agreement between the parents is to find the percent of times that each of them agree on the personality description. Agreements are shown in the diagonal cells (cells 11, 22 and 33) in Table 3. In this example 88 parents agree on "1", 40 parents agree on "2" and 12 parents agree on "3". So, 140 out of 200 parents, or 70% agree on the description. But, we don't know if this is good or not, because we don't know what the level of agreement would be "by chance." The "by chance" levels of agreement are given by the expected counts for those cells. The expected counts are found in the same manner that we found expected frequencies for Chi Square,

E = (row total*column total)/N.

In the above example the sum of the expected counts in the diagonal cells (cells 11, 22, and 33) gives us the expected frequency of agreement (60+18+4=82) . So, now we can compare the observed levels of agreement with the levels of agreement expected by chance. Kappa provides the means for doing this. The formula for kappa is

k = (Oa - Ea)/(N - Ea)

where Oa is the observed count of agreement, Ea is the expected count of agreement, and N is the total number of respondent pairs.

k = (Oa - Ea)/(N - Ea)
   = (140 - 82)/(200 - 82) = 58 / 118 = .492

Kappa is the proportion of agreements after chance agreement has been excluded. Its upper limit is +1.00 (total agreement). If judges agree at a chance level, k = 0.00. The lower limit of kappa depends on the distribution of row and column marginals and can fall between 0 and -1.00. (Normally we are interested in levels of agreement greater than chance rather than smaller than chance.)

If you are interested in additional reading about kappa see Cohen (1960) or Kraemer (1982).

top

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46.

Kraemer, H. C. (1982). Kappa coefficient. In S. Kotz and N. L. Johnson (Eds.), Encyclopedia of Statistical Sciences. New York: John Wiley & Sons.