Remove Duplicate Lines
Strip repeated lines from a list, with case and whitespace options.
Remove Duplicate Lines tool
What this tool does
Paste a list and get it back with the repeats gone and the original order intact. That last
part matters: sort | uniq is the usual shell answer and it reorders everything,
which is wrong when the list is a changelog, a queue or a ranked set of results.
The second mode inverts the question and returns only the values that appeared more than once. That is the faster way to answer "what is duplicated in this export?", and it is usually the reason someone opened a deduplicator in the first place.
Common uses
- Cleaning a mailing list where the same address was collected from three forms, often with different capitalisation.
- Finding duplicate IDs in a CSV column before a unique-constraint violation finds them for you.
- Collapsing a repetitive log file down to its distinct messages to see what is actually happening.
- Merging two lists of dependencies, tags or feature flags without hand-checking the overlap.
A short example
With "ignore case" on, this input:
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]becomes three lines, and the report says two values were repeated:
[email protected]
[email protected]
[email protected]
Switch to "show only duplicates" and the same input returns
[email protected] and [email protected] — one copy of each repeated
value, not every copy of it.
What "duplicate" means here
By default, two lines are duplicates only when they are byte-for-byte identical. That is strict on purpose, because whitespace and capitalisation are meaningful in code and in file paths. Three toggles loosen it:
- Ignore case compares in lowercase. Correct for email addresses, whose domain part is case-insensitive by specification; wrong for Linux file paths.
- Trim before comparing strips leading and trailing whitespace before the comparison but keeps it in the output. This is what catches the trailing space that a spreadsheet export leaves on every third row.
- Keep first vs keep last chooses which copy survives. Keep last is the one you want when later lines are the corrected versions — an audit log, or an append-only settings file where the final value wins.
Worth knowing
A trailing newline in your input produces a final empty line, and two of them count as duplicates of each other. Enable "drop blank lines" if that skews the count. Invisible characters are the other common surprise: two lines that look identical but differ by a zero-width space or a non-breaking space are genuinely different strings and will both survive. If the numbers here do not match what your eyes say, run the text through the extra-space remover first.
Frequently asked questions
Why does uniq leave duplicates in my file?
uniq collapses only runs of adjacent identical lines, so any repeat that is not sitting next to its twin survives — which is why it is nearly always written as sort | uniq, or replaced by sort -u. The order-preserving equivalent of this page is awk '!seen[$0]++' file: it keeps the first occurrence of every line and streams the rest through untouched.
Why do identical-looking lines survive when the file came from Windows?
Windows ends each line with a carriage return plus a line feed. Mix that file with Unix-formatted text and every line from it secretly carries a trailing U+000D, so value and the carriage-returned value are genuinely different strings and both are kept. Switching on trim before comparing removes it, because a carriage return is whitespace; normalising line endings first does the same job permanently.
How do I drop rows duplicated by one column rather than the whole line?
This page compares complete lines, so a CSV export where only the email column repeats looks entirely unique — the timestamps differ. Paste just that column, or key on a field directly with awk -F, '!seen[$2]++' file.csv. If the copy you want to keep is the newest, order the rows by date in the line sorter first and set this tool to keep the last copy.