Intraclass Correlation Calculator

How to Use the Intraclass Correlation Calculator

1. 1. Paste the ratings. One subject per row, one rater or occasion per column — copied from a spreadsheet, or typed with tabs, commas or spaces. A header row is optional.
2. 2. Pick the form. Choose the ICC that matches your design: the model, absolute agreement or consistency, and one rater's score or the average. The table underneath always shows all six.
3. 3. Set the confidence level. 95% is the convention; 90% and 99% are there if a protocol asks.
4. 4. Read and report. You get the coefficient, its interval, the F test of ICC = 0, a Koo and Li label and the ANOVA table. Copy result gives one line in the form journals expect.

Which ICC Should You Use?

There is no single "the ICC". Shrout and Fleiss (1979) described six, and McGraw and Wong (1996) renamed them by what they measure. Three questions pick the one you need:

• Did the same raters rate every subject? If not — each subject was scored by a different set of raters — use the one-way model, ICC(1,1) or ICC(1,k). If yes, use a two-way model.
• Do the raters matter beyond this study? If they are a sample of all possible raters and you want the result to generalise to them, the raters are random: ICC(2,1) or ICC(2,k). If these particular raters are the only ones of interest, they are fixed: ICC(3,1) or ICC(3,k).
• Must raters agree in level, or only in ranking? Absolute agreement counts a rater who scores everyone two points higher as disagreeing; consistency does not. Koo and Li (2016) advise absolute agreement for test–retest and intra-rater studies, where a systematic shift is a real change, and for inter-rater studies when any rater could be used in practice.

Use a single-measure form when one rater's score will be used in practice, and an average-measure form when the mean of all k raters will. Average-measure values are always larger, so say which you computed. A two-way mixed model with absolute agreement has the same point estimate as ICC(2,1); only the interpretation differs.

The Formulas

All six come from a two-way ANOVA of subjects × raters, with mean squares MSR (between subjects), MSC (between raters), MSE (residual) and MSW (within subjects, the pooled MSC and MSE), n subjects and k raters:

The average-measure forms are ICC(1,k) = (MSR − MSW)/MSR, ICC(2,k) = (MSR − MSE)/(MSR + (MSC − MSE)/n) and ICC(3,k) = (MSR − MSE)/MSR. Confidence intervals are F-based: Shrout and Fleiss's limits for the one-way and consistency forms, McGraw and Wong's Satterthwaite degrees of freedom for absolute agreement, and the Spearman–Brown step kρ / (1 + (k − 1)ρ) to turn single-measure limits into average-measure ones. On the Shrout and Fleiss data this reproduces the point estimates and single-measure limits printed in the psych::ICC documentation.

Worked Example: Six Targets, Four Judges

Shrout and Fleiss's own data — six targets rated by four judges — load with Load example. Judge 2 scores far lower than the others, which is exactly what separates the absolute-agreement and consistency forms.

The mean squares are MSR = 11.24, MSC = 32.49, MSE = 1.02 and MSW = 6.26. Because MSC is large, the judges differ in level; ICC(2,1) = 0.290 counts that as disagreement, while ICC(3,1) = 0.715 ignores it and sees only that the judges rank the targets alike. Both F tests give F(5, 15) = 11.03, p < .001. Reported in APA style: ICC(2,1) = .29, 95% CI [.02, .76], F(5, 15) = 11.03, p < .001.

Interpreting and Reporting an ICC

Koo and Li (2016) suggest reading the confidence interval, not the point estimate: below 0.5 is poor reliability, 0.5–0.75 moderate, 0.75–0.9 good and above 0.9 excellent. An estimate of 0.82 with an interval from 0.55 to 0.94 spans moderate to excellent — the sample is too small to say which. Report the form by name ("two-way random-effects, absolute-agreement, single-rater ICC(2,1)"), the number of subjects and raters, and the interval.

• Reporting "the ICC" without a form. ICC(2,1) and ICC(3,1) can differ by more than 0.4, as above.
• Using a consistency form for agreement. If raters must give the same score, not just the same ranking, use absolute agreement.
• Treating a restricted range as a weak instrument. An ICC is a ratio of variances; if all subjects are alike, between-subject variance is small and the ICC falls even when raters are precise.
• Using the ICC for categories. For nominal or ordinal categories use Cohen's or Fleiss' kappa; for continuous measurements compared with a reference method, a Bland–Altman plot shows the size of the differences, which the ICC does not.

Frequently Asked Questions

What is a good intraclass correlation coefficient?

Koo and Li (2016) classify ICC values as poor below 0.5, moderate from 0.5 to 0.75, good from 0.75 to 0.9 and excellent above 0.9, and apply those bands to the 95% confidence interval's bounds rather than to the estimate alone.

What is the difference between ICC(2,1) and ICC(3,1)?

Both use the same ANOVA, but ICC(2,1) is an absolute-agreement coefficient that treats differences in raters' average level as error, while ICC(3,1) is a consistency coefficient that ignores them. When some raters score systematically higher, ICC(2,1) is lower. Choose by whether raters must give the same score or merely rank subjects the same way.

How do I choose between single and average measures?

Use single measures if one rater's score will be used for each subject in practice, and average measures if the mean of k raters will be. The average form is always the larger number, so quoting it when single ratings will be used overstates reliability.

How many subjects and raters do I need?

Koo and Li suggest at least 30 heterogeneous samples and at least 3 raters as a rule of thumb, but what matters is the interval you need. Check the confidence interval this calculator returns — a wide one means too few subjects.

Can the ICC be negative?

Yes. When variation between raters or within subjects exceeds variation between subjects, the estimate falls below zero. That means reliability is essentially zero (the true value cannot be negative), so report it as such and look at the interval.

What does the calculator do with missing ratings?

The ANOVA formulas need a complete subjects × raters table, so any subject with a missing rating is left out, and the page says how many were dropped. If many are missing, estimate the variance components with a mixed model instead.