How do I remove duplicate lines but keep the original order?

Keep a set of lines already seen and output each line only the first time it appears. This tool does that, so the result is in the original order; the Unix sort -u command, by contrast, sorts the lines first.

What is the difference between uniq and sort -u?

uniq only removes adjacent duplicates, so it is normally run after sort. sort -u sorts and deduplicates in one step. Neither preserves the original order; a set-based approach like this tool does.

Why are two lines that look identical not treated as duplicates?

They probably differ in invisible characters: trailing spaces, tabs, a non-breaking space, or letter case. Turn on "Ignore leading and trailing spaces" and "Ignore case", or check the lines in a hex viewer.

Duplicate Line Remover

Keep each line once.

About Duplicate Line Remover

A duplicate line remover keeps the first occurrence of every line and drops the repeats, with options to ignore case or spaces and keep blank lines.

How to use

Paste the lines. The unique lines appear on the right as you type.
Adjust the options: "Ignore case" and "Keep blank lines" are off by default, "Ignore leading and trailing spaces" is on.
Read the counts of lines in, lines out and removed, then Copy or Download unique-lines.txt.

Options

Option	Default	Effect
Ignore case	off	"Apple" and "apple" count as the same line
Ignore leading and trailing spaces	on	" apple" and "apple" count as the same line
Keep blank lines	off	Blank lines are removed; when on, the first blank line is kept and the rest are removed as duplicates

Order is preserved: the first occurrence stays where it was and later copies are removed.
The kept line is written out exactly as it appeared, even when "Ignore case" or the trim option decided it was a duplicate.
Lines are split on \n; a Windows \r\n file is handled because the trim option removes the trailing \r before comparing.
Does not sort the output, count occurrences or remove duplicates within a line.

Example

apple
banana
apple
cherry
→
apple
banana
cherry
Lines in 4 · Lines out 3 · Removed 1

Privacy

Deduplication runs in the browser. No request is made, and nothing is stored.

Frequently asked questions

How do I remove duplicate lines but keep the original order?: Keep a set of lines already seen and output each line only the first time it appears. This tool does that, so the result is in the original order; the Unix sort -u command, by contrast, sorts the lines first.
What is the difference between uniq and sort -u?: uniq only removes adjacent duplicates, so it is normally run after sort. sort -u sorts and deduplicates in one step. Neither preserves the original order; a set-based approach like this tool does.
Why are two lines that look identical not treated as duplicates?: They probably differ in invisible characters: trailing spaces, tabs, a non-breaking space, or letter case. Turn on "Ignore leading and trailing spaces" and "Ignore case", or check the lines in a hex viewer.

Encode & decode

GUID Generator: One or many UUIDs, version 4 or 7.
ULID Generator: Sortable 26-character IDs with a time prefix.
Password Generator: Random passwords with length and character-set controls.
Hash Generator: MD5, SHA-1, SHA-256, SHA-384, SHA-512 and HMAC for text or a file.
Base64 Encoder / Decoder: Base64, Base32 and Base58 in both directions.
ROT13 / Caesar Cipher: Shift letters by any amount, or ROT13 in one click.
JWT Decoder: Read a token’s header and payload without sending it anywhere.
URL Encoder / Decoder: Percent-encode text for URLs, or decode it back.
HTML Entities: Escape &, < and > for HTML, or unescape them.
Base64 Image Encoder: Image to a Base64 data URL for CSS or HTML, or decode one back.
Random Number Generator: Random integers in a range, one or many, with or without repeats.
Bcrypt Generator: Hash a password with bcrypt, or check one against a hash.
AES Encrypt / Decrypt: Encrypt text with a passphrase using AES-256-GCM, or decrypt it.
Morse Code Translator: Text to dots and dashes, and back.

Data formats

XML ⇄ JSON: Convert between XML and JSON, from text or a file.
YAML ⇄ JSON: Convert between YAML and JSON, from text or a file.
CSV ⇄ JSON: Convert between CSV and JSON, from text or a file.
JSON to TypeScript: Infer TypeScript interfaces from a JSON sample.
JSON to Python Dataclass: Infer Python dataclasses from a JSON sample.
JSON to Pydantic Model: Infer Pydantic models from a JSON sample.
JSON to Go Struct: Infer Go structs with json tags from a JSON sample.
JSON to Rust Struct: Infer Rust structs with serde derives from a JSON sample.
JSON to Kotlin: Infer Kotlin data classes from a JSON sample.
JSON to C# Class: Infer C# classes from a JSON sample.
Base Number Converter: Binary, octal, decimal, hex and any base up to 36.
Text ⇄ Binary / Hex: Text to binary, hex, octal or decimal, and back.
Unix ⇄ UTC: Epoch seconds or milliseconds to a readable date, and back.
Timezone Converter: Convert a time between cities, with DST handled.
Cron Converter: Describe a schedule in words and get the cron expression.
Cron Expression Parser: Explain a cron expression in words and list its next run times.
Days Between Dates: The exact time between two dates, in days, weeks, months and years.
Age Calculator: Exact age in years, months and days, and the next birthday.
Epoch Clock: The current Unix timestamp, live, in seconds and milliseconds.
CIDR / Subnet Calculator: Network, broadcast, mask and host count for an IPv4 CIDR block.
Chmod Calculator: File permissions as octal (755) or symbolic (rwxr-xr-x).

Units & measures

Angle: Degrees, radians, gradians and turns.
Area: Square metres, feet, acres and hectares.
Bits / Bytes: Bits, bytes and every prefix up to petabytes.
Energy: Joules, calories, kilowatt-hours and BTU.
Frequency: Hertz, kilohertz, megahertz and RPM.
Fuel Economy: Miles per gallon, km per litre and litres per 100 km.
Length: Metres, feet, miles, inches and nautical miles.
Power: Watts, kilowatts and horsepower.
Pressure: Pascals, bar, PSI and atmospheres.
Speed: km/h, mph, knots and metres per second.
Temperature: Celsius, Fahrenheit and Kelvin.
Time Unit: Seconds, minutes, hours, days and beyond.
Volume: Litres, gallons, pints and cubic metres.
Weight: Kilograms, pounds, ounces and stone.

Text & code

Lorem Ipsum Generator: Placeholder paragraphs, sentences or words.
Slug Generator: Turn a title into a URL-safe slug.
Text Compare: Diff by line, word or character, unified or side by side.
URL Parser: Break a URL into host, path, query and fragment.
JSON Formatter & Validator: Validate, pretty-print or minify JSON.
JSON Diff: Compare two JSON documents, ignoring key order and formatting.
JSONPath Tester: Query JSON with a JSONPath expression.
SQL Formatter: Format and indent SQL for any dialect.
JavaScript Beautifier: Format minified or messy JavaScript into readable code.
HTML Formatter: Indent and tidy HTML, or strip it down to one line.
XML Formatter: Pretty-print or minify XML, with error positions.
CSS Minifier / Beautifier: Minify CSS, or expand it back to readable rules.
HTML to Markdown: Convert HTML to clean Markdown.
Text Case Converter: camelCase, snake_case, kebab-case, Title Case and more.
Markdown Previewer: Render Markdown as you type.
Whitespace Remover: Strip line breaks, extra spaces or both.
Sort Lines: Sort lines alphabetically, numerically, by length or reversed.
Duplicate Line Remover: Keep each line once.
String Counter: Characters, words, lines and bytes.
Regex Tester: Test a pattern against text with live matches.
Regex Cheatsheet: Regular-expression syntax with examples.

Word play

Palindrome Checker: Does it read the same backwards?
String Reverser / Rotator: Reverse a string or rotate it by n characters.
Roman Numeral Converter: 1984 ⇄ MCMLXXXIV.
NATO Phonetic Alphabet: Spell text out as Alfa, Bravo, Charlie.
Pig Latin Converter: English to Pig Latin and back.
Numeronym Generator: internationalization → i18n.
Leetspeak Translator: Text to l33t and back.
Word Frequency Counter: Which words appear most, and how often.

Images & colour

QR Code Generator: QR codes for text, links, Wi-Fi and contacts.
Barcode Generator: EAN, UPC, Code 128 and more as SVG or PNG.
Barcode / QR Reader: Decode a barcode or QR code from an image.
Color Picker: Pick a colour and copy it in every format.
Image Converter: Convert between PNG, JPEG and WebP, and resize on the way.
Contrast Checker: Contrast ratio of text on a background, against WCAG AA and AAA.
Favicon Generator: Every favicon size from one image.
Image Compressor: Shrink a JPEG, PNG or WebP in the browser.
Gradient Generator: Build a CSS gradient and copy the rule.
Box Shadow Generator: Tune a shadow and copy the CSS.

Reference

ASCII Table: Every ASCII character with its decimal, hex, octal, binary and HTML code.
HTTP Status Codes: Every HTTP status code with what it means and when to use it.
MIME Types: Common MIME types by file extension, searchable.
Port Numbers: Well-known and registered TCP and UDP ports and the services behind them.