Regex to remove duplicate lines

This pattern finds a line that is immediately repeated one or more times. Replacing the whole match with the first line, written $1, leaves a single copy. A backreference makes the pattern compare each line with the next.

The pattern

/^(.*)(?:\r?\n\1)+$/gm

The tester opens with this pattern and the examples below already loaded. Nothing you type is uploaded.

How it works

^(.*)
Capture a whole line into group 1.
\r?\n
A line break (Windows or Unix style).
\1
Backreference: the next line must be identical to group 1.
(?:...)+$
Repeat for every following identical line, up to the end of a line.

Examples

Matches

  • apple apple
  • a a a
  • same line same line same line

Does not match

  • apple banana
  • apple apples
  • one two three

Use it in JavaScript

// Adjacent duplicates only
const a = text.replace(/^(.*)(?:\r?\n\1)+$/gm, '$1');

// All duplicates, keeping first occurrence and order
const b = [...new Set(text.split(/\r?\n/))].join('\n');

Limits and tips

  • A regex only removes duplicates that sit next to each other. For a, b, a both a lines stay. Sort the lines first, or use the Set method below, to remove all duplicates.
  • The comparison is exact and case-sensitive, and trailing spaces count. Trim lines first if "Apple " and "Apple" should be treated as the same.
  • In the tester, set the replacement to $1. In JavaScript pass "$1" to replace().
  • For very large inputs the Set approach is faster and easier to reason about than backreferences.

Frequently asked questions

Can a regex remove all duplicate lines, not just adjacent ones?

Not reliably. Use [...new Set(text.split(/\r?\n/))].join("\n"), which keeps the first occurrence of each line in the original order.

What does $1 mean in the replacement?

$1 inserts whatever group 1 captured, here the first copy of the line, so the repeated lines collapse into it.