Cohen's Kappa Calculator
Measure how well two raters agree beyond chance. Paste a k × k agreement table or two columns of category labels and get Cohen's kappa with its standard error, 95% confidence interval and z test — plus linear and quadratic weighted kappa when the categories are ordered. Free, no sign-up, and nothing leaves your browser.
A square table of counts, pasted from a spreadsheet or typed with tabs, commas or spaces. Category names in the first row and column are optional.
Paste an agreement table, or two columns of ratings, to see Cohen's kappa.
How to Use the Cohen's Kappa Calculator
- 1. Choose your data format. Use Agreement table if you already have counts — rater A's categories down the side, rater B's across the top. Use Two columns of ratings if you have one row per subject with each rater's label.
- 2. Paste the data. Copy straight from a spreadsheet or type it with tabs, commas or spaces. Category names in a header row are optional. The calculator shows the agreement table it built, with row and column totals, so you can check it.
- 3. Set the category order for ordinal ratings. Weighted kappa treats categories as ordered from low to high. Numeric codes sort automatically; for words such as Fail, Pass and Merit, type the order in the box.
- 4. Read kappa and its interval. You get κ, its 95% confidence interval, observed and chance agreement, and a z test of κ = 0. With three or more categories, a table adds linear and quadratic weighted kappa. Copy the APA-style line when you are done.
The Cohen's Kappa Formula
Kappa compares the agreement the raters actually reached with the agreement two raters would reach by chance if they assigned categories independently at their own observed rates:
Formula
κ = (pₒ − pₑ) / (1 − pₑ)
- pₒ — observed agreement: the share of subjects both raters put in the same category (the diagonal of the table divided by n).
- pₑ — chance agreement: Σ (row total × column total) / n², the agreement expected if each rater kept their own category proportions but chose independently.
- κ = 1 means perfect agreement, κ = 0 means agreement no better than chance, and negative values mean less agreement than chance.
Weighted kappa gives partial credit for near-misses on an ordered scale: κw = (pₒ(w) − pₑ(w)) / (1 − pₑ(w)), where every cell is multiplied by an agreement weight. Linear (Cicchetti–Allison) weights are w = 1 − |i − j| / (k − 1); quadratic (Fleiss–Cohen) weights are w = 1 − (i − j)² / (k − 1)². With quadratic weights, kappa is closely related to the intraclass correlation.
The 95% confidence interval is κ ± 1.96 × SE, using the large-sample standard error of Fleiss, Cohen and Everitt (1969); the z test of κ = 0 uses a second standard error computed under that null hypothesis. Both are asymptotic, so treat them as approximate in small samples. They are the formulas SAS PROC FREQ uses, and the calculator reproduces its documented examples to four decimals.
Worked Example: Two Teachers Grading 50 Essays
Two teachers each rated the same 50 essays Low, Medium or High. Press Load example in table mode to enter their counts:
| Low | Medium | High | Total | |
|---|---|---|---|---|
| Low | 14 | 3 | 1 | 18 |
| Medium | 2 | 13 | 4 | 19 |
| High | 0 | 2 | 11 | 13 |
| Total | 16 | 18 | 16 | 50 |
The teachers agree on 14 + 13 + 11 = 38 essays, so pₒ = 38 / 50 = 0.76. Chance agreement is pₑ = (18 × 16 + 19 × 18 + 13 × 16) / 50² = 838 / 2,500 = 0.3352. Then κ = (0.76 − 0.3352) / (1 − 0.3352) = 0.639, with SE = 0.091, 95% CI [0.461, 0.817], and z = 6.41, p < .001. With these margins the largest kappa possible is 0.910, so the ceiling is not what holds kappa down here — the 12 disagreements are.
| Weights | κ | SE | 95% CI | z |
|---|---|---|---|---|
| Unweighted | 0.639 | 0.091 | [0.461, 0.817] | 6.41 |
| Linear | 0.699 | 0.081 | [0.541, 0.857] | 6.40 |
| Quadratic | 0.762 | 0.078 | [0.610, 0.914] | 5.43 |
The weighted kappas are higher because 11 of the 12 disagreements are only one category apart. Because Low, Medium and High are ordered, a weighted kappa is a fair summary here; for unordered categories, such as diagnoses, only the unweighted value makes sense.
How to Interpret and Report Kappa
| Kappa | Strength of agreement |
|---|---|
| Below 0 | Poor |
| 0.00–0.20 | Slight |
| 0.21–0.40 | Fair |
| 0.41–0.60 | Moderate |
| 0.61–0.80 | Substantial |
| 0.81–1.00 | Almost perfect |
Landis and Koch described their divisions as arbitrary, and other authors use stricter ones, so give the number and its interval rather than the label alone. Kappa also depends on the margins. When one category is very common, chance agreement is high and kappa can be low even when the raters agree on most subjects — the "kappa paradox" described by Feinstein and Cicchetti (1990). Reporting pₒ, pₑ and the maximum attainable kappa alongside κ helps a reader see why.
An APA-style report of the example: "Agreement between the two teachers was substantial, Cohen's κ = .64, 95% CI [.46, .82], z = 6.41, p < .001 (n = 50)." Kappa cannot exceed 1, so APA style drops the leading zero.
Common Mistakes
- Reporting percent agreement as reliability. 76% agreement sounds strong, but two teachers grading independently at their own rates would agree on about 34% of these essays by chance; kappa removes that share.
- Weighting nominal categories. Weighted kappa assumes an order. For categories with no order, use the unweighted value.
- Mixing up the table. Rows must be one rater and columns the other, with the categories in the same order on both sides, so that agreements fall on the diagonal.
- Testing κ = 0 and stopping there. A significant z only says agreement beats chance — a weak kappa can be highly significant in a large sample.
Frequently Asked Questions
- What is a good Cohen's kappa value?
- On Landis and Koch's (1977) benchmarks, 0.41–0.60 is moderate, 0.61–0.80 substantial and above 0.80 almost perfect. What is acceptable depends on the stakes: ratings that feed into decisions about individuals, such as diagnoses, need much higher agreement than exploratory coding. Because the benchmarks are conventions, report kappa with its confidence interval and compare it with what similar studies in your field report.
- What is the difference between Cohen's kappa and percent agreement?
- Percent agreement is simply pₒ, the share of subjects the raters classified the same way. Kappa subtracts the agreement expected by chance and rescales the result, so 0 means chance-level agreement and 1 means perfect agreement. Two raters who both label 90% of cases "normal" will agree often by chance alone, which percent agreement rewards and kappa does not.
- When should I use weighted kappa, and which weights?
- Use weighted kappa when the categories are ordered, so that confusing Low with Medium is a smaller error than confusing Low with High. Linear weights penalise disagreement in proportion to the distance between categories; quadratic weights penalise distant disagreements more heavily and give values close to an intraclass correlation. Say which weights you used, because they change the result.
- Why is kappa low when the raters agree most of the time?
- Because chance agreement is high when one category dominates. If both raters call 95% of cases negative, they will agree on most cases even when rating at random, so pₑ is large and the room above it is small. This is the kappa paradox; report pₒ and pₑ alongside κ, and the calculator's maximum attainable kappa, so readers can see the effect of the margins.
- Can I use Cohen's kappa with more than two raters?
- No — Cohen's kappa compares exactly two raters. When each subject is rated by the same number of raters, use Fleiss' kappa. Some studies with a fixed panel of raters instead average the Cohen's kappa of every pair of raters (Light's kappa).