Text
Remove duplicate lines
Paste a list — log lines, email addresses, SKUs, tags — and get back only the unique ones. Order is preserved by default, so you can still see which entry came first.
How to use it
- Paste or type your list, one item per line.
- Press “Keep first” to de-duplicate while preserving order.
- Press “Sort + de-duplicate” if you want an alphabetical result.
- Copy the output with the button above it.
Three kinds of de-duplication
Keeping the first occurrence preserves order, which is right for logs, timelines and any list where sequence carries meaning. Keeping the last occurrence is right when later entries supersede earlier ones, as with config overrides or repeated imports. Sorting before de-duplicating produces a canonical list, which is what you want before diffing two exports. This tool offers the first and the third, because between them they cover most real work.
Normalise before comparing
Most duplicates that survive a de-duplication pass differ invisibly: a trailing space, a non-breaking space pasted from a web page, or a different Unicode normalisation form of the same accented letter. Trim trailing whitespace first, and if the data arrived from several sources, normalise to NFC before comparing. Otherwise you will be told a list has forty unique values when it really has thirty-eight.
A mailing-list cleanup, start to finish
Export the list, paste it in, de-duplicate keeping the first occurrence, then count what is left against the original. A list of 12,000 addresses that drops to 9,400 is not a bug — it is usually the same person signing up from a work and a personal address, or a re-import.
Clean the cheap mismatches before comparing: trailing spaces, mixed case in the domain part (the local part is technically case-sensitive, but in practice nobody means it that way), and addresses wrapped in angle brackets from a forwarded message. Do that first and the duplicate count usually doubles.
Frequently asked questions
Does it ignore case?
No — “Apple” and “apple” are treated as different lines by default, which is what most lists of filenames and IDs need. Normalise case first with the Case Converter if you want them merged.
Are blank lines removed?
Blank lines count as a value, so all but the first one disappear. That is usually the desired result for pasted lists that end with stray newlines.
How large a list can it handle?
Tens of thousands of lines are fine. The work happens in your browser, so the limit is your device's memory rather than a server quota.
Is my data sent anywhere?
No. There is no upload step — the text never leaves the tab.
Does it handle duplicates that differ only in case?
Not by default, because Apple and apple are different strings. Lowercase the list first if you want them merged.
Is empty-line removal included?
No — an empty line is a value. Remove blank lines as a separate pass so you can see how many there were.