Regex to remove duplicate lines
This pattern finds a line that is immediately repeated one or more times. Replacing the whole match with the first line, written $1, leaves a single copy. A backreference makes the pattern compare each line with the next.
The pattern
/^(.*)(?:\r?\n\1)+$/gmThe tester opens with this pattern and the examples below already loaded. Nothing you type is uploaded.
How it works
^(.*)- Capture a whole line into group 1.
\r?\n- A line break (Windows or Unix style).
\1- Backreference: the next line must be identical to group 1.
(?:...)+$- Repeat for every following identical line, up to the end of a line.
Examples
Matches
apple applea a asame line same line same line
Does not match
apple bananaapple applesone two three
Use it in JavaScript
// Adjacent duplicates only
const a = text.replace(/^(.*)(?:\r?\n\1)+$/gm, '$1');
// All duplicates, keeping first occurrence and order
const b = [...new Set(text.split(/\r?\n/))].join('\n');Limits and tips
- A regex only removes duplicates that sit next to each other. For a, b, a both a lines stay. Sort the lines first, or use the Set method below, to remove all duplicates.
- The comparison is exact and case-sensitive, and trailing spaces count. Trim lines first if "Apple " and "Apple" should be treated as the same.
- In the tester, set the replacement to $1. In JavaScript pass "$1" to replace().
- For very large inputs the Set approach is faster and easier to reason about than backreferences.
Frequently asked questions
Can a regex remove all duplicate lines, not just adjacent ones?
Not reliably. Use [...new Set(text.split(/\r?\n/))].join("\n"), which keeps the first occurrence of each line in the original order.
What does $1 mean in the replacement?
$1 inserts whatever group 1 captured, here the first copy of the line, so the repeated lines collapse into it.