Delphi Consensus Calculator
Paste each Delphi round's ratings — a row per panelist, a column per item — and get the median, interquartile range, mean, and percent agreement and disagreement for every item, then test them against the consensus rules you choose: a percent-agreement threshold, an IQR limit, a median floor, and stability between rounds. It runs in your browser and nothing is uploaded. Free, no sign-up.
Changing the scale resets these to common starting points. Set them to whatever your protocol states.
Consensus rules
There is no single definition of consensus in Delphi studies. Switch on the rules your protocol set in advance; an item reaches consensus when every switched-on rule is met.
Quartiles — and so the IQR — depend on the method, most of all in small panels. Report which you used.
Rounds
Paste one table per round: a row per panelist and a column per item, in the same item order every round. Blank cells are missing ratings.
Results
Paste at least one round of ratings to see the statistics.
How to Use the Delphi Consensus Calculator
- 1. Set the scale and the bands. Choose the rating scale, then say which ratings count as agreement (for example 4 and 5 on a five-point scale) and which count as disagreement.
- 2. Switch on your rules. Tick the consensus rules your protocol fixed before round 1 and set their thresholds. An item has consensus when every ticked rule is met.
- 3. Paste the rounds. One table per round, with panelists in rows and items in columns, in the same item order each time. Blank cells count as missing, and ratings off the scale are left out and reported.
- 4. Read the results. Each round shows n, median, IQR, mean, percent agree and disagree, and a result with the unmet rules named. A second table follows each item across rounds, and you can copy or download everything.
There Is No Single Definition of Consensus
The Delphi method asks a panel to rate items over several rounds, feeding back the group's results between rounds so panelists can reconsider. What it does not provide is a standard for when the group agrees. A systematic review of 100 Delphi studies (Diamond et al., 2014) found that about 98 claimed to assess consensus, only 72 defined it, and only 42 of the studies that concluded consensus was achieved had set a threshold in advance. The authors' conclusion was that definitions vary widely and are poorly reported. This tool therefore applies your definition and shows its working, rather than imposing one.
| Rule | Common convention | Where it comes from |
|---|---|---|
| Percent agreement | 70–80% of panelists rating in the agree band; 75% is the usual choice | Most common definition in Diamond et al. (2014), median threshold 75% |
| Interquartile range | IQR ≤ 1 on a 5-point scale | A widely used convention; criteria reviewed in von der Gracht (2012) |
| Median | Median at or above a floor, such as 4 on a 5-point scale, for items to be kept | Study-specific inclusion rule |
| Stability | Little change in the share agreeing between rounds | Study-specific; set your own limit |
Treat the first two rows as conventions, not standards: they are what many studies chose, and a protocol that picks different values can be perfectly sound if it states them first. Diamond et al. also found that only 23 studies tied stopping to achieving consensus, while 70 simply stopped after a set number of rounds.
Why the Quartile Method Matters
The IQR is Q3 minus Q1, and software disagrees about how to find a quartile. For the ratings 1, 2, 3, 4, 5 the inclusive method (Excel QUARTILE.INC, R's default) gives an IQR of 2, while the exclusive method (QUARTILE.EXC, SPSS) and the median-of-halves method both give 3. With a panel of 10 to 20 on a five-point scale, that is the difference between meeting an IQR ≤ 1 rule and missing it. Choose one method, state it in your protocol, and report it.
A Worked Example
Press Load example: 12 panelists rating six course components from 1 to 5 over two rounds. Agreement is a 4 or 5, and the rules are at least 75% agreement and an IQR of 1 or less. In round 1 three items reach consensus. Preregistration does not: its median is 3, its IQR is 1.25, and 33% agree. In round 2 it has a median of 4, an IQR of 0.25, and exactly 75% agree, so it just meets the threshold. Reading primary papers goes from 67% to 92%. Qualitative methods stays out: only 8% agree, and 33% disagree, so the direction is towards disagreement but short of the 75% needed. By round 2, five of the six items are in consensus.
Reporting Consensus
- State the rules before round 1. The scale, the agree band, the threshold, and any rule on the IQR, median, or stability.
- Report each round. Panelists invited and responding, items rated, and items reaching consensus, with the median and IQR or percent agreement for each item.
- Name the quartile method if you report an IQR.
- Say how you stopped. Whether you stopped on consensus, on stability, or after a planned number of rounds.
References
- Diamond, I. R., Grant, R. C., Feldman, B. M., Pencharz, P. B., Ling, S. C., Moore, A. M., & Wales, P. W. (2014). Defining consensus: A systematic review recommends methodologic criteria for reporting of Delphi studies. Journal of Clinical Epidemiology, 67(4), 401–409. https://doi.org/10.1016/j.jclinepi.2013.12.002
- von der Gracht, H. A. (2012). Consensus measurement in Delphi studies: Review and implications for future quality assurance. Technological Forecasting and Social Change, 79(8), 1525–1536. https://doi.org/10.1016/j.techfore.2012.04.013
What this calculator does not do
It summarizes ratings and applies rules; it does not design the study, choose the panel, or run the rounds. Agreement among panelists is not evidence that they are right, and a high percent agreement on a small panel is fragile. Report the panel size next to every percentage.
Frequently Asked Questions
- What is consensus in a Delphi study?
- There is no single definition. Studies usually define it as a minimum percent agreement (often about 75%), a maximum interquartile range, or both, and sometimes require stability across rounds. Whatever you choose, set the definition before the first round and report it.
- What percent agreement counts as consensus in a Delphi?
- Thresholds between 70% and 80% are common, and in the systematic review by Diamond et al. (2014) the median threshold among studies using percent agreement was 75%. That is a convention, not a rule. Choose a value that suits your scale and panel size, and justify it.
- What does an IQR of 1 or less mean for consensus?
- On a five-point scale the middle half of the ratings spans at most one scale point, so most panelists are close together. It is a widely used convention. On a longer scale the same limit is stricter, so set the cut-off to suit your scale; it is yours to define.
- How many rounds does a Delphi study need?
- Two or three rounds are typical, but the number should follow your protocol. Stop when your stated criteria are met or when ratings stop changing, or after a pre-planned number of rounds, and say which. Diamond et al. found most studies stopped after a set number of rounds rather than on consensus.
- How do I cite Diamond et al. (2014) in APA?
- Diamond, I. R., Grant, R. C., Feldman, B. M., Pencharz, P. B., Ling, S. C., Moore, A. M., & Wales, P. W. (2014). Defining consensus: A systematic review recommends methodologic criteria for reporting of Delphi studies. Journal of Clinical Epidemiology, 67(4), 401–409. https://doi.org/10.1016/j.jclinepi.2013.12.002
- How do I handle panelists who skip an item?
- Leave the cell blank. The item's statistics use the ratings it received, and n is shown beside them so a reader can see how many panelists stand behind a percentage. Report non-responses per round in your methods.
- Does this tool store my panel's ratings?
- No. The calculation runs in your browser and nothing is uploaded. If your ratings are identifiable, use panelist codes rather than names in the first column.