1

I have a file with some lines (incredible, right :) and need to clean it up. I would like to delete all but the first instance of the line. Is there a plugin for Vim that lets me select the line (visual line selection) that keeps the first instance of the line, and delets all further ones, regardless of whether I selected one of the further ones (for I may miss the first instance at the beginning of the file)?

If you have an idea of how to quickly select the line, delete every but the first instance in a file, that would work too.

All ideas on this welcomed.

4
  • 1
    Are the identical lines sequential? Are they scattered all over the file?
    – seliopou
    Jan 9, 2013 at 13:55
  • Do you care about the order of the lines? If not, you can do ":sort u". Jan 9, 2013 at 15:38
  • @JiříPospíšil - Unfortunatelly :), the order is important.
    – Rook
    Jan 9, 2013 at 18:45
  • @seliopou - Scattered all over the file, yes.
    – Rook
    Jan 9, 2013 at 18:45

4 Answers 4

3

I hope you have awk available. If true, you can do in this way:

  • open your file in vim
  • type (in one line) :command! RML exec '%!awk -vl="'.getline(".").'" ''$0==l{if(\!f)print $0;f=1;next;}1'''

this will create a command called RML (Remove Line)

  • then move your cursor to the line you want to remove duplicates, type :RML<ENTER> the duplicates are removed, and only first line stays.

  • this way will keep the order of lines in your original file

  • this way works no matter your repeated line are continued or not

for detail you could check out the gif screencast below:

enter image description here

1
  • I don't have awk available. In addition I would like to incorporate this line deletion as a part of the function, and therefore would like to depend on other programs as little as possible.
    – Rook
    Feb 21, 2013 at 11:38
0

maybe simpler without vim :

if the line that you want to keep only 1 occurence of matches the regexp "aregexp",

grep "$aregexp" file | head -n 1 > /tmp/thatline #first occurence of the line.
all_other_occurences="$(fgrep -n "$(cat /tmp/thatline)" file | head -n +2 | tac)"
   #fgrep string file : search string in file, literrally, without using regexp
   #head -n +2 (or on some machines: head -n +1) skips the 1st line of the input
   #tac : so that we have line numbers in reverse order (allowing to delete them without moving the next ones up!)
#here add a test to see if all_other_occurence is empty : nothing to do then (there was no lines other than the first one)
#otherwise: 
for linetodelete in "$all_other_occurence" ; do
    # either delete line n in file using sed -i,
    #    or     create&concatenate a vim command that later will be used to
    #           delete those lines from inside vim, via script or copy/paste manually...
done

another approach (probably faster!): delete all lines and reinsert the 1st occurence at the right place

thelinenumber=$( awk '$0~var{print NR;exit}' var="$aregexp" < file)
   #will exit as soon as it finds the 1st occurence
theline="$(sed -e "${thelinenumber}p" < file)" 
fgrep -v "$theline" file > newfile #delete all occurence
awk -v l="$thelinenumber" -v s="$theline" 'NR == l {print s} {print}' file > file.new

I did not check those, but I hope you can see the intent and correct any syntax problem (and add double checks, for empty variables, the test if there is just 1 occurence of the line, etc)

1
  • I am still looking for a vim solution.
    – Rook
    Feb 21, 2013 at 12:09
0

a quick way is this

go to the line you want

ma
0y$
:g/\V<ctrl-r>"/m'a
:/\V<ctrl-r>"/d
0

My PatternsOnText plugin provides a

:DeleteDuplicateLinesOf .

command (as well as other related commands) that does exactly that.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.