Duplicate Word Remover
Paste your text and pick a mode. Consecutive repeats turns a typo such as "the the" into "the" without touching the rest of your prose; All duplicates keeps only the first copy of every word in a word list. You see how many words were removed and how many remain. Free, no sign-up, and the text never leaves your browser.
Mode
0 words
How to Use the Duplicate Word Remover
- Choose a mode. Pick Consecutive repeats only for sentences and paragraphs, or All duplicates in a word list for keywords, tags and vocabulary.
- Paste your text. The counter under the box shows how many words you have pasted.
- Click Remove Duplicates. The cleaned text appears below with two cards: Duplicate Words Removed and Words Remaining.
- Check, then copy. Read the result once before you use it, because a few repeats are deliberate (see below). Copy Result puts it on your clipboard.
What Each Mode Does
Consecutive repeats only looks for the same word twice or more in a row, separated only by spaces or tabs on one line, and keeps a single copy. The comparison ignores capitalisation, so "The the" collapses to "The" — the first copy keeps its own capitalisation. A run of any length is reduced to one word. Hyphens and apostrophes count as part of a word, and the match works for accented Latin letters and for Cyrillic, Arabic, Devanagari and other scripts that put spaces between words.
It is deliberately cautious. A word repeated across a line break, a repeat with punctuation between the copies ("very, very"), and a repeat glued to a digit ("the the2") are all left alone, because any of them might be intentional.
All duplicates in a word list is for lists, not prose. It splits every line on spaces and tabs, compares the words in lower case, and keeps only the first occurrence of each one, wherever the later copies sit. Line breaks and blank lines you typed are preserved, but a line whose every word was a repeat is dropped rather than left empty. Punctuation counts as part of the word, so "tips," and "tips" are different entries.
Worked Examples
Consecutive repeats on a 20-word sentence with four stuttered words and one three-in-a-row:
Input
I went to to the the store and and bought bread. The the cat sat sat sat on the mat.
Output
I went to the store and bought bread. The cat sat on the mat.
Six copies were removed (the three-in-a-row counts as two) and 14 words remain. Note that the unrelated "the" before "mat" survives: only back-to-back repeats are collapsed. Now the list mode on a short word list:
Input
red green blue Red green yellow blue purple
Output
red green blue yellow purple
Three words were removed — Red, the second green and the second blue — leaving five. The first copy of each word stays where it was, so the second line shrinks to "yellow".
Repeats That Are Not Mistakes
English has legitimate doubled words: "She had had enough", "the idea that that author raised", "no, no, no". Consecutive mode will collapse the first two, so scan the output before you paste it into a final draft. Running the mode on a list or a poem with deliberate repetition would also change its meaning.
The same caution applies far more strongly to list mode. Run on ordinary prose, it removes every later "the", "and" and "a": "The cat saw the dog and the cat ran." becomes "The cat saw dog and ran." Use it only on lists.
Which Tool Do You Need?
| If you want… | Use |
|---|---|
| To fix "the the" typos in a paragraph, or dedupe a list of words | This page |
| To dedupe one item per line, with a case option and blank-line removal | Remove Duplicate Lines |
| An alphabetised list of every distinct word in a text, with counts | Unique Word Extractor |
| To find words repeated close together but not adjacent | Word Repetition Checker |
| The items two lists share and the items only one has | List Comparison Tool |
Frequently Asked Questions
- How do I remove duplicate words from text?
- Paste the text, choose a mode and click Remove Duplicates. Use Consecutive repeats only to collapse words that are doubled back to back, and All duplicates in a word list to keep just the first copy of every word in a list. For running prose, use the first mode.
- How do I fix a doubled word like "the the"?
- Choose Consecutive repeats only and paste the passage. Every back-to-back repeat on the same line, in any capitalisation, is reduced to one word, and the count shows how many were removed.
- Does it keep the first or the last copy?
- The first. In both modes the first occurrence keeps its original spelling and capitalisation. In list mode the first "Red" would stay even if a lower-case "red" came later, because the comparison ignores case.
- Why was "very, very" left in my text?
- A comma, full stop or other mark between two words stops them counting as consecutive. The tool errs on the side of leaving text alone, since emphatic repetition is common. If you do want it gone, delete the extra word by hand.
- What happens to a line when all its words are duplicates?
- List mode drops a line when every word on it was already seen earlier, so one-word-per-line lists do not leave gaps. Blank lines you typed yourself are kept. For one item per line, Remove Duplicate Lines compares whole lines and can strip blank lines too.
- Does it work for languages other than English?
- Yes, for any language that separates words with spaces, including accented Latin text and Cyrillic, Arabic and Devanagari. It cannot split Chinese or Japanese, which have no spaces between words, so a whole run of characters is treated as a single word.
- Is my text sent anywhere?
- No. The words are compared in your browser and the text is not uploaded or stored.