Remove Duplicate Rows

About removing duplicate rows

The text version of this job compares whole lines, and for a list of email addresses that is exactly right. A spreadsheet needs something different.

In a real table, duplicates are almost never defined by the whole row. Two exports of the same order are the same order even though one has a later timestamp; two records for the same customer are the same customer even though the address was tidied in between. What makes a row a duplicate is one or two columns, not all of them.

So the key columns are the first control here, not a hidden setting. Tick the ones that decide what "the same" means, and everything else is allowed to differ.

The choices that matter

• Which columns — all of them to start with. Untick the ones that are allowed to vary, such as a timestamp or a note field
• Keep the first or the last — this decides which version survives when the other columns differ. Keeping the last is usually right for an append-only export, where the newest row is the corrected one
• Ignore extra spaces — on by default. " Acme Ltd" and "Acme Ltd" are the same company to everyone except a string comparison
• Match case — off by default, so ACME and Acme are treated as one

Tick Show the removed rows instead to see exactly what would be dropped. Nothing is thrown away silently: the count is always on screen, and the removed rows are one click away.

How to use it

• Paste your rows or upload a spreadsheet
• Untick any column that is allowed to differ between duplicates
• Choose whether the first or the last copy survives
• Check the count, and glance at the removed rows if the number surprises you
• Download as CSV or Excel, or copy it straight back

Duplicates that do not look like duplicates

The rows that survive a deduplication when you expected them to go are usually not a bug in the matching. They are two strings that look identical on screen and are not identical in memory.

The usual culprits are invisible. Text copied from a web page often carries a non-breaking space instead of an ordinary one — same width, different character. A CSV that came out of an older system may have a trailing space on every value in a column. And a number typed with a leading apostrophe in Excel is text, not a number, so 1000 and 1000 can genuinely differ.

Ignore extra spaces handles the common half of this: it trims the ends and collapses runs of whitespace, and it treats a non-breaking space as a space. What it deliberately does not do is strip punctuation or normalise accents, because Muller and Müller are different people often enough that guessing would be worse than leaving it to you.

If two rows still refuse to match, paste both into a plain text editor and look at the ends of the values. The difference is nearly always there.

What people use it for

• Cleaning a mailing list that was built from several exports
• Removing repeated orders or transactions after a double import
• Collapsing a customer table where the same account was entered twice
• Tidying a survey export where people submitted more than once
• Finding out how many duplicates there actually are before deciding anything
• Getting a genuine unique count for a report

Good to know

• Works on Windows, macOS, Linux, ChromeOS, and on phones and tablets — it takes pasted text or a file, not a folder
• Nothing is uploaded. Reading and deduplicating both happen in the page
• Row order is preserved. The surviving rows come back in the order they arrived, not sorted
• Untick every column and it falls back to matching on the whole row, which is the safe interpretation of "no columns chosen"
• Numbers are compared as they are written. If one sheet has 1000 and another 1,000 as text, they are different — use Compare Two Excel Files if that distinction is what you are chasing
• The preview shows the first 200 rows; the download always contains every one

Related tools

• Remove Duplicate Lines — the same job for a plain list of text
• Compare Two Excel Files — see what differs rather than removing it
• Group and Sum — collapse duplicates by totalling them instead
• Compare Lists — what is in one list and not the other