Remove Duplicate Lines
Keeps the first occurrence of each line, original order preserved
About Remove Duplicate Lines
Removes duplicate lines from a list, keeping the first occurrence of each and leaving everything else in its original order. Useful for cleaning up email lists, URL dumps and log files.
The three options matter more than they look. With "Ignore case" on, Apple and apple count as the same line. With "Trim spaces" on, " abc" and "abc" count as the same line — data copied out of spreadsheets and web pages often carries invisible leading spaces, and without this option a batch of duplicates slips through. "Drop blank lines" is on by default because duplicated blank lines are meaningless anyway.
When it finishes it tells you how many lines were removed. That number is the quickest way to check whether the deduplication actually did anything.
Frequently asked questions
Does it change the order of my lines?
No. Each line stays where it first appeared; only later duplicates are removed.
Are lines with different capitalisation treated as duplicates?
Not by default. Turn on "Ignore case" to treat them as the same line.
What about lines that differ only by leading or trailing spaces?
They count as duplicates by default, because "Trim spaces" is on. Turn it off to compare exactly.
How many lines can it handle?
Up to about 100,000 lines without trouble — everything runs locally in your browser, nothing is uploaded.
