Multiple Comparisons Correction Calculator

How to Use the Multiple Comparisons Correction Calculator

1. 1. Enter the p-values. Paste them separated by commas, spaces or new lines, or switch to “Labelled” and give each test a name (“Age effect, 0.012”). Entries like “p = .03” and “< .001” are read; a “< .001” is taken as .001.
2. 2. Set α and the family size. The default α is 0.05. If your family has more tests than the p-values you listed, enter the total; the unlisted tests are treated as p = 1, exactly as R’s p.adjust does with its n argument.
3. 3. Compare the adjusted p-values. Every method’s adjusted p-value appears beside the raw one, and a star marks those at or below α. Compare adjusted values with α directly — no further arithmetic needed.
4. 4. Pick a method before you look. The comparison table says what each method controls, and the step table shows Holm and Benjamini–Hochberg rank by rank.

Why Correct for Multiple Comparisons?

Each test run at α = 0.05 has a 5% chance of a false positive when nothing is going on. Run several and the chances compound: for m independent tests the probability of at least one false positive is 1 − (1 − α)^m. That is 22.6% for 5 tests, 40.1% for 10 and 64.2% for 20. Multiple-comparison corrections shrink the per-test threshold, or equivalently inflate the p-values, so that the error rate for the whole family stays where you want it.

There are two targets. The family-wise error rate (FWER) is the probability of any false positive, which is the strict standard for confirmatory work. The false discovery rate (FDR) is the expected proportion of rejected hypotheses that are false, which suits exploratory work with many tests — genomics, screening, large batteries of outcomes — where a few false leads are acceptable.

The Methods

The step procedures differ in direction. Holm starts with the smallest p-value and stops at the first one it cannot reject; Hochberg and Benjamini–Hochberg start with the largest and reject everything up to the first one that passes. Adjusted p-values are made monotone, so a larger raw p-value never gets a smaller adjusted one.

A Worked Example

Click “Try an example” for ten p-values: 0.001, 0.008, 0.039, 0.041, 0.042, 0.06, 0.074, 0.205, 0.212 and 0.216. Five are below 0.05 before correction. With m = 10, Bonferroni multiplies by 10, giving adjusted values 0.01, 0.08, 0.39 and so on, so only the smallest survives. Holm gives 0.01, 0.072, 0.312, … and also keeps only one. Benjamini–Hochberg multiplies by m ÷ rank — for the second p-value, 10 ÷ 2 × 0.008 = 0.04 — and then takes the smallest value from that rank onward, giving 0.01, 0.04, 0.084, 0.084, 0.084. At α = 0.05 it keeps two.

The labelled example uses six tests: Reading 0.049, Writing 0.03, Math 0.8, Science 0.011, History 0.004 and Art 0.0003. Five are below 0.05 raw. Bonferroni keeps two (adjusted 0.0018 and 0.024), Holm keeps three (0.0018, 0.02, 0.044), and Benjamini–Hochberg keeps four (0.0018, 0.012, 0.022, 0.045). The more liberal the method, the more it finds, which is why the choice has to be made before the data are seen.

Choosing and Reporting

For a small confirmatory family, use Holm: it needs no assumptions and beats Bonferroni. Use Benjamini–Hochberg when you have many tests and can tolerate a known fraction of false discoveries, and Benjamini–Yekutieli if you cannot assume the tests are independent or positively dependent. Say which you used and the family: “p-values were adjusted with the Holm procedure across the six subject tests.” Report adjusted p-values (or say the threshold was adjusted), and keep the raw ones available. For all-pairs comparisons after an ANOVA, a purpose-built method such as Tukey’s HSD is usually better than Bonferroni.

Frequently Asked Questions

What is the Bonferroni correction?

It multiplies each p-value by the number of tests m (capping at 1), or equivalently tests each at α ÷ m. With 10 tests at α = 0.05 the per-test threshold becomes 0.005. It guarantees the family-wise error rate stays at or below α under any dependence, at the cost of power.

What is the difference between Holm and Bonferroni?

Holm is a step-down version of Bonferroni. It tests the smallest p-value at α ÷ m, the next at α ÷ (m − 1), and so on, stopping at the first failure. It controls the same error rate with no extra assumptions but is never less powerful, so there is rarely a reason to prefer plain Bonferroni.

When should I use Benjamini–Hochberg instead of Bonferroni?

When you run many tests and a few false positives are an acceptable price for finding more true effects, as in gene-expression studies or exploratory screens. Benjamini–Hochberg controls the false discovery rate, not the chance of any error, so it is much more powerful. For a handful of confirmatory tests, stay with a family-wise method.

What is an adjusted p-value?

The smallest significance level at which a hypothesis would be rejected after accounting for the whole family of tests. You can compare it directly with α. An adjusted p-value of 0.044 means the result is significant at α = 0.05 but not at 0.04.

Do these results match R and other software?

Bonferroni, Holm, Hochberg, Hommel, Benjamini–Hochberg and Benjamini–Yekutieli follow R’s p.adjust (methods bonferroni, holm, hochberg, hommel, BH and BY), including its n argument for a larger family. Šidák and Holm–Šidák are not in p.adjust; they follow their textbook formulas, as in GraphPad and SAS.

Do I need to correct for multiple comparisons?

Whenever you will draw a conclusion from any one of several tests on the same question or data. Some analysts treat a single pre-specified primary test as outside the family, but exploratory sets of tests are not. Whatever you decide, state the family and the method in your methods section.