> ## Content Index
> Fetch the complete content index at: https://mfitzp.ghost.io/llms.txt
> Use this file to discover other available public pages before exploring further.

# Remove duplicate lines from text files (with sort)
- URL: https://mfitzp.ghost.io/remove-duplicate-lines-from-text-files-with-sort/
- Published: 2012-01-08T00:00:00.000Z
- Updated: 2023-06-26T09:30:03.000Z
- Author: Martin Fitzpatrick
- Tags: Tutorials, Bash, #Import 2024-03-14 08:20, #Import 2026-08-24 14:26

A quick method to remove duplicates from text files - including for example CSV files - where multiple records have been added (perhaps automatically) at different times resulting in multiple copies of the same record scattered throughout the file. Here is a simple one-liner bash command to remove duplicates using sort.

🐧

This method is sensitive to the line endings of the file. If you have files editing in a combination of Unix/Linux/Mac/Windows you may have a variety of line-endings in place.  
  
The simplest route is to run the file through `dos2unix` before attempting the sort/unique filter.

In a bash shell enter:

```Bash
sort -u file.csv -o file.csv

```

This takes your file, sorts it (using sort), gets the unique entries `-u` and writes it to the outfile `-o` which here is the same as the initial file.