Krippendorff's Alpha Calculator
Paste your coders' ratings — any number of coders, with gaps wherever a coder skipped a unit — and get Krippendorff's alpha for nominal, ordinal, interval and ratio data side by side, with the coincidence matrix, the observed and expected disagreement, and an optional bootstrap confidence interval. It follows Krippendorff's 2011 computing guide and reproduces its published worked example. Free, no sign-up, and nothing leaves your browser.
Paste from a spreadsheet, or separate values with tabs, commas or spaces. Words are fine for nominal data; ordinal, interval and ratio need numbers. A header row is optional.
Nominal: categories with no order. Ordinal: ordered categories. Interval: equal steps, no true zero. Ratio: equal steps with a true zero.
Paste your coders' ratings to see Krippendorff's alpha for each level of measurement.
How to Use the Krippendorff's Alpha Calculator
- 1. Paste the ratings. One unit (a tweet, an essay, a patient) per row and one coder per column, copied from a spreadsheet or typed with tabs, commas or spaces. Leave a cell blank or write NA when a coder did not rate that unit. Switch the layout if your coders are in rows instead.
- 2. Choose the level of measurement. Nominal for categories with no order, ordinal for ordered categories, interval for equal steps, ratio for equal steps with a true zero. The table underneath shows all that your data allow.
- 3. Read alpha. You get α, the number of units and pairable values, and Krippendorff's reliability label. The coincidence matrix shows exactly which values were paired.
- 4. Add an interval if you need one. The bootstrap button resamples the units 2,000 times and gives a 95% interval and the chance that α is below 0.667 or 0.800.
What Krippendorff's Alpha Measures
Alpha compares the disagreement the coders actually showed with the disagreement you would expect if they had assigned the same values at random: α = 1 − Dₒ / Dₑ. A value of 1 means perfect agreement, 0 means the coders agree no more than chance, and negative values mean systematic disagreement. It is built to work in every situation where kappa breaks down: two or many coders, different numbers of ratings per unit, missing data, and four levels of measurement.
The calculation starts from the coincidence matrix. Every unit's ratings are paired in every order, and each pair counts 1/(m − 1), where m is how many ratings the unit received, so units with more coders do not dominate. Units with fewer than two ratings cannot be compared and are set aside. Dₒ averages the squared difference between paired values, Dₑ averages it over all pairs in the data, and the squared difference depends on the metric:
| Metric | Squared difference between c and k | Use it for |
|---|---|---|
| Nominal | 0 if c = k, otherwise 1 | Categories: topic codes, yes/no |
| Ordinal | Cumulative frequency distance between c and k | Ordered categories: Likert items, severity grades |
| Interval | (c − k)² | Scores with equal steps: test marks, temperatures |
| Ratio | ((c − k) / (c + k))² | Counts and measurements with a true zero: lengths, durations |
Worked Example: Four Coders, Twelve Units
Press Load example for the reliability matrix from Krippendorff's 2011 guide — four coders, twelve units, and seven missing ratings. One unit has only a single rating, so eleven units and 40 pairable values enter the calculation.
| Metric | α | Dₒ | Dₑ |
|---|---|---|---|
| Nominal | 0.743 | 0.200 | 0.779 |
| Ordinal | 0.815 | 47.275 | 256.077 |
| Interval | 0.849 | 0.433 | 2.872 |
| Ratio | 0.797 | 0.022 | 0.111 |
For nominal data α = 1 − 0.200 / 0.779 = 0.743. Treating the values as ordered or numeric gives credit for near-misses (a 2 and a 3 are closer than a 1 and a 4), which is why the other metrics are higher. The 95% bootstrap interval for the nominal α is 0.426 to 1.000 — with only eleven units, even this apparently good agreement is poorly pinned down. These are the values the paper publishes, and the same matrix is the example in the help page of R's irr package.
Interpreting and Reporting Alpha
Krippendorff (2004) recommends relying on variables with α ≥ 0.800, drawing only tentative conclusions where 0.667 ≤ α < 0.800, and discarding data below 0.667. These are conventions, not laws: choose the cut-off from the cost of acting on a wrong code. Report the metric, α, the number of units and coders, and an interval — for example, "Krippendorff's α (ordinal) = 0.815, 95% bootstrap CI [0.459, 1.000], 11 units." The copy button writes that line.
- Choosing the wrong metric. Numeric codes for unordered categories (1 = politics, 2 = sport) must be analyzed as nominal; interval treats 1 and 3 as farther apart than 1 and 2.
- Skewed categories. When one value dominates, expected disagreement is small and alpha can be low even though raw agreement is high. Report the distribution of codes alongside alpha.
- Too few units. The point estimate hides how uncertain it is; use the bootstrap interval and avoid labelling a variable reliable from a handful of units.
- Dropped units. A unit rated by only one coder says nothing about reliability, so it is excluded; check the note that tells you how many were set aside.
Frequently Asked Questions
- What is a good Krippendorff's alpha?
- Krippendorff recommends α ≥ 0.800 as reliable and 0.667 ≤ α < 0.800 as good enough only for tentative conclusions. Values below 0.667 mean the coding is too unreliable to use. These cut-offs are guidelines — a high-stakes decision justifies a stricter threshold.
- How is Krippendorff's alpha different from Cohen's and Fleiss' kappa?
- Cohen's kappa handles exactly two coders and Fleiss' kappa needs the same number of coders on every unit. Krippendorff's alpha allows any number of coders, missing ratings, and ordinal, interval and ratio data as well as nominal. For two coders, nominal data and no missing values it is nearly identical to Scott's pi.
- How does the calculator handle missing data?
- A blank cell or NA is simply not a rating. Each unit contributes only the ratings it has, weighted so that units with more ratings do not count extra. A unit with fewer than two ratings is set aside because nothing can be compared; the page says how many.
- Which level of measurement should I choose?
- Match the metric to what the codes mean. Use nominal if the categories have no order, ordinal if they are ranked but the gaps are unequal, interval if the gaps are equal, and ratio if zero means none of the quantity. Ordinal, interval and ratio need numeric values.
- Why is my alpha low when the coders agree most of the time?
- Alpha corrects for the agreement expected by chance. If nearly every unit gets the same value, chance agreement is already very high and a few disagreements pull alpha down sharply. Look at the coincidence matrix and the spread of the codes before deciding the coders are at fault.
- How is the bootstrap interval calculated?
- The calculator draws the same number of units as you have, with replacement, recomputes alpha, and repeats 2,000 times; the 2.5th and 97.5th percentiles of those alphas are the interval. It resamples whole units and uses a fixed random seed, so the same data always give the same interval. It is a percentile bootstrap, not the specialised pair-resampling routine of Hayes and Krippendorff's macro.