Fleiss' Kappa Calculator
Measure agreement among three or more raters who sort subjects into categories. Paste a subjects × categories table of counts — or raw ratings, one row per subject — and get Fleiss' kappa, the observed and chance agreement, a z test, and a kappa for each category, with the working shown. Free, no sign-up, and it runs entirely in your browser.
Each cell is how many raters put that subject in that category, so every row adds up to the number of raters. Subject names in the first column and category names in a header row are optional.
Paste a table of counts, or one row of ratings per subject, to see Fleiss' kappa.
How to Use the Fleiss' Kappa Calculator
- 1. Pick a data format. Counts per category takes one row per subject and one column per category, each cell saying how many raters chose that category. One rating per cell takes the raw data — one row per subject, one column per rater — and counts it for you.
- 2. Paste the data. Copy straight from a spreadsheet. Subject names in the first column and category names in a header row are optional. In the counts table, every row must add up to the same number of raters; any that do not are highlighted.
- 3. Check the table. The calculator shows the counts it will use, with each subject's total in the n column, before giving any result.
- 4. Read the results. You get Fleiss' κ, the mean observed agreement P̄, chance agreement P̄ₑ, a z test of κ = 0, and a κ for each category. Copy the APA-style line for your write-up.
The Fleiss' Kappa Formula
Fleiss (1971) extended kappa to any number of raters. Suppose N subjects are each rated by n raters into k categories, and nᵢⱼ raters put subject i in category j. Agreement within one subject is the share of rater pairs that agree:
Formulas
Pᵢ = (Σⱼ nᵢⱼ² − n) / (n(n − 1))
P̄ = (1/N) Σᵢ Pᵢ P̄ₑ = Σⱼ pⱼ² κ = (P̄ − P̄ₑ) / (1 − P̄ₑ)
- P̄ — the mean agreement across subjects.
- pⱼ — the share of all N × n ratings that fell in category j.
- P̄ₑ — the agreement expected if raters chose categories at random in those overall proportions.
- κ — agreement beyond chance, scaled so that 1 is perfect and 0 is chance level.
The calculator also gives κⱼ for each category, which treats that category against all the others combined, and a z test of κ = 0 using the standard error of Fleiss, Nee and Landis (1979). That standard error is derived under the null hypothesis, so it is valid for the test but not for a confidence interval. For an interval, bootstrap the subjects.
Worked Example: Four Coders, Ten Posts
Four coders classified ten social-media posts as Positive, Neutral or Negative. Press Load example to enter the counts:
| Post | Positive | Neutral | Negative | Pᵢ |
|---|---|---|---|---|
| 1 | 4 | 0 | 0 | 1.000 |
| 2 | 3 | 1 | 0 | 0.500 |
| 3 | 0 | 4 | 0 | 1.000 |
| 4 | 0 | 3 | 1 | 0.500 |
| 5 | 1 | 1 | 2 | 0.167 |
| 6 | 0 | 0 | 4 | 1.000 |
| 7 | 2 | 2 | 0 | 0.333 |
| 8 | 0 | 1 | 3 | 0.500 |
| 9 | 4 | 0 | 0 | 1.000 |
| 10 | 0 | 4 | 0 | 1.000 |
For post 2, Pᵢ = (3² + 1² + 0² − 4) / (4 × 3) = 6 / 12 = 0.500. The ten values average to P̄ = 0.700. The 40 ratings split 14 Positive, 16 Neutral and 10 Negative, so pⱼ = 0.35, 0.40 and 0.25, and P̄ₑ = 0.35² + 0.40² + 0.25² = 0.345. Then κ = (0.700 − 0.345) / (1 − 0.345) = 0.542, moderate agreement on Landis and Koch's benchmarks. With SE₀ = 0.0922, z = 5.88 and p < .001.
The category kappas show where the coders disagree: Positive κ = 0.634, Negative κ = 0.556 and Neutral κ = 0.444. Neutral is the category the coders find hardest to apply consistently — a cue to sharpen its definition in the codebook before coding the full sample.
Interpreting and Reporting Fleiss' Kappa
Fleiss' kappa is usually read against Landis and Koch's (1977) benchmarks: 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial and above 0.80 almost perfect. As with Cohen's kappa, the cut-offs are conventions, and kappa is pulled down when one category dominates the ratings even if agreement is high. Report κ with the number of subjects, raters and categories, and the category kappas when they differ.
In APA style: "Agreement among the four coders was moderate, Fleiss' κ = .54, z = 5.88, p < .001 (10 subjects, 4 raters each)." Kappa cannot exceed 1, so the leading zero is dropped.
Assumptions and Common Mistakes
- Unequal numbers of ratings. Every subject needs the same number of raters. Subjects with missing ratings must be dropped, or use Krippendorff's alpha, which allows them.
- Ordinal categories. Fleiss' kappa treats categories as unordered, so a near-miss counts the same as a wide miss. For ordered ratings, consider an intraclass correlation.
- Rows that do not add up. In the counts format a row is how many raters chose each category, not which ones, so each row must total n.
- Reading the z test as a measure of strength. A tiny p-value only says agreement beats chance; the size of κ says how well the raters agree.
- Expecting Cohen's kappa with two raters. With two raters, Fleiss' kappa equals Scott's pi, which can differ slightly from Cohen's kappa.
Frequently Asked Questions
- What is the difference between Fleiss' kappa and Cohen's kappa?
- Cohen's kappa measures agreement between exactly two raters and uses each rater's own category proportions to estimate chance agreement. Fleiss' kappa handles any number of raters and pools all the ratings to estimate chance. With two raters, Fleiss' kappa reduces to Scott's pi rather than to Cohen's kappa, so the two can differ slightly.
- Do the same raters have to rate every subject?
- No. Fleiss' kappa only needs every subject to receive the same number of ratings; different people can supply them. That suits designs where, say, any three of twenty reviewers code each document. If one fixed panel rates everything, Fleiss' kappa still works, though some researchers prefer averaging Cohen's kappa over every pair of raters.
- What is a good Fleiss' kappa value?
- On Landis and Koch's (1977) benchmarks, 0.41–0.60 is moderate, 0.61–0.80 substantial and 0.81–1.00 almost perfect agreement. Those labels are conventions rather than tests, and what is acceptable depends on how the ratings will be used. Report the value with its context — subjects, raters, categories — rather than the label alone.
- Why doesn't the calculator give a confidence interval for Fleiss' kappa?
- The standard error Fleiss, Nee and Landis (1979) derived assumes κ = 0, so it is correct for testing whether agreement beats chance but too small or too large for building an interval around a non-zero kappa. A bootstrap over subjects — resample the rows, recompute κ, take the 2.5th and 97.5th percentiles — gives a defensible interval.
- Can I use Fleiss' kappa for ordinal ratings?
- You can, but it ignores the order: rating a subject 1 instead of 2 counts as the same disagreement as rating it 1 instead of 5. For ordered scales, an intraclass correlation or Krippendorff's alpha with an ordinal metric uses that information.
- What should I do if some subjects have missing ratings?
- Fleiss' kappa requires the same number of ratings per subject. In the raw-ratings format, the calculator keeps the subjects with the most common number of ratings and tells you how many it left out. If many subjects are incomplete, Krippendorff's alpha is designed for missing ratings.