Remove duplicate lines
When the same line appears more than once, keeps only the first occurrence.
Example
- Options used
- Ignore case✓
Input
Seoul Busan seoul Seoul·· Incheon Busan
Output
Seoul Busan Incheon
Symbols in examples: ⇥ tab · · trailing space · ⍽ NBSP · □ full-width space · ◌ zero-width character
Open this operation in the toolWhat it does
Reads lines from the top and deletes any line identical to one already seen. The first occurrence stays and the original order is preserved. No sorting is needed, so it also works on lists where order matters.
Options
- Ignore case: treats
Appleandappleas the same line. Ideal for email address lists. The line that remains keeps the spelling of its first occurrence. - Ignore leading, trailing and repeated spaces: on by default. Only for comparison, surrounding spaces are removed and repeated spaces count as one, so differences in spacing do not hide duplicates. The spaces in the output lines are not changed, so put Trim leading and trailing spaces first if you also want them cleaned.
- Leave blank lines alone: when on, blank lines are excluded from the duplicate check and all of them are kept. When off, only one blank line remains.
When to use it
Use it to remove people who appear in several merged lists, to see each repeated log message only once, or to tidy a keyword list. You can see how many lines were removed from the change in line count shown on the pipeline step.
If the values are comma-separated, first spread them one per line with Add line breaks (split).