8

I have got text file:

1 2 4 5 6 7
1 3 5 6 7 8
1 2 3 4 5 6
1 2 4 5 6 7

Here first and last line are simmilar. I have a lot of files that have double lines. I need to delete all dublicate.

0

5 Answers 5

12

All these seem really complicated. It is as simple as:

gc $filename | sort | get-unique > $output

Using actual file names instead of variables:

gc test.txt| sort | get-unique > unique.txt
0
6

To get unique lines:

PS > Get-Content test.txt | Select-Object -Unique
1 2 4 5 6 7
1 3 5 6 7 8
1 2 3 4 5 6

To remove the duplicate

PS >  Get-Content test.txt | group -noelement | `
      where {$_.count -eq 1} | select -expand name

1 3 5 6 7 8
1 2 3 4 5 6
2
  • Caveat (just in case if ...): Select-Object -Unique is case sensitive and this is not configurable. Sep 30, 2011 at 13:49
  • Good point and so is Get-Unique. I'm able to work around it with sort-object: 'one two three','one two Three','one Two three','one Two four' | sort -unique
    – Shay Levy
    Sep 30, 2011 at 14:07
1

If order is not important:

Get-Content test.txt | Sort-Object -Unique | Set-Content test-1.txt

If order is important:

$set = @{}
Get-Content test.txt | %{
    if (!$set.Contains($_)) {
        $set.Add($_, $null)
        $_
    }
} | Set-Content test-2.txt
0

Try something like this:

$a = @{} # declare an arraylist type
gc .\mytextfile.txt | % { if (!$a.Contains($_)) { $a.add($_)}} | out-null

$a #now contains no duplicate lines

To set the content of $a to mytextfile.txt:

$a | out-file .\mytextfile.txt
0
0
$file = "C:\temp\filename.txt"
(gc $file | Group-Object | %{$_.group | select -First 1}) | Set-Content $file

The source file now contains only unique lines

The already posted options did not work for me for some reason

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.