Page & Bell

Duplicate Line Remover

Paste any list and duplicate lines disappear instantly — with full control over what counts as a duplicate. Match case-insensitively, trim whitespace, or ignore punctuation before comparing, while the output always keeps your original line text untouched. A single-pass algorithm handles 100,000+ lines without breaking a sweat, entirely in your browser.

Matching options (output keeps the original line text)

Paste a list above — duplicates are removed instantly.

How to use the duplicate line remover

  1. Paste your list, one item per line — emails, keywords, URLs, SKUs, anything.
  2. Choose how duplicates are matched: case-insensitive, trim before compare, ignore punctuation. These affect matching only; output lines keep their original text.
  3. Pick whether to keep the first or last occurrence of each duplicate, and how to treat blank lines.
  4. Read the stats bar (total / unique / removed), then copy the result or download it as a .txt file.
  5. Flip on “show only the duplicates” to audit what would be removed instead.

Where deduping saves you real trouble

Mailing-list uploads. Most email platforms (Mailchimp, Brevo, Zoho Campaigns) count contacts toward your billing tier and some flag accounts that repeatedly import duplicates. Deduping with case-insensitive + trim before upload catches the classic Name@Gmail.com vs name@gmail.com pair that exact-match importers treat as two people.

Keyword imports. Google Ads rejects duplicate keywords within an ad group and Google Keyword Planner exports routinely contain the same term in different cases. Dedupe here, then feed the clean list to the Keyword Combiner or convert it to a comma-separated row with the Comma Separator.

Auditing instead of deleting. The inverse mode answers a different question: not “give me the clean list” but “what exactly is duplicated?” Run it on a SKU export and you get one line per value that appears two or more times — the shortlist to investigate in your source system.

Worked example

Input (6 lines): Apple, apple  (trailing space), Banana, banana!, Cherry, Apple.

  • Default options (trim only): 5 unique — the two exact Apple lines merge; case and punctuation keep the rest apart.
  • + case-insensitive: 4 unique — apple joins the Apple group.
  • + ignore punctuation: 3 unique — banana! joins Banana. Stats bar reads: 6 total / 3 unique / 3 removed.

If you want to know how often each value occurred rather than just removing repeats, the Word Frequency Counter gives you counts and percentages instead.

Frequently asked questions

Do the matching options change my actual lines?

No. Case-insensitive, trim, and ignore-punctuation affect only the comparison key. If your list contains “John@Example.com ” and “john@example.com”, case-insensitive + trim treats them as one duplicate, but the surviving line is output exactly as you pasted it (the first or last occurrence, your choice). This matters for data like emails where you want dedupe logic to be loose but the stored value to stay verbatim.

When should I keep the last occurrence instead of the first?

Keep-last is the right choice when later entries are more up to date — for example a change log of customer records where each re-entry supersedes the previous one, or a re-exported keyword list where the newest line carries corrected spelling. Keep-first preserves your original ordering priority; keep-last preserves recency. The kept line is emitted at the position of that occurrence.

How is this better than Excel’s Remove Duplicates?

Three ways: you don’t need a spreadsheet open for a 10-second job; Excel compares cells exactly as stored, so “Apple ” with a trailing space and “apple” survive as three “different” values unless you build helper columns with TRIM and LOWER; and Excel has no ignore-punctuation mode at all. Here those normalizations are one checkbox each, and the stats bar tells you exactly how many rows were dropped — Excel only tells you after the fact.

Will it catch near-duplicates like plurals or reordered words?

No — and be suspicious of any tool that silently claims to. “running shoe” vs “running shoes” vs “shoe running” are distinct lines to an exact-match deduper, even with all normalizations on. Catching those requires stemming and token sorting, which changes meaning in ways you should review by hand. For keyword lists, sort the output A–Z first: near-duplicates cluster together visually, making manual review fast.

How are blank lines handled?

Blank lines never participate in duplicate matching — an empty line will not “dedupe” against another empty line unless you ask. The blank-line policy gives you three behaviours: keep them all (default, preserves paragraph grouping), collapse runs of blanks to a single one, or remove every blank line for a tight list ready to upload.

What is the size limit and how fast is it?

The deduper does one pass with a hash map: normalize each line once, look it up, move on — O(n) overall. A 100,000-line list (roughly 2–3 MB) processes in well under a second in a modern browser. A trailing newline at the end of your paste is ignored, so it never shows up as a phantom blank line in the counts, and Windows CRLF endings are normalized automatically.

Related tools