0

I have a file containing tens of thousands of lines containing duplicates. I want to find the mean time/distance between the duplicates in terms of line numbers.

For example: (where the first column is a line number)

1 string1
2 string2
3 string2
4 string1
5 string3

would give 2 (3 lines between the first pair of duplicates, 1 line between the second duplicate, divided by 2 since there are 2 duplicates).

Any ideas on how to approach this?

EDIT

Starting test!
32-bit hash: 0x995D9A6E
32-bit hash: 0xA27B264D
32-bit hash: 0x856ED0A5
32-bit hash: 0x3B83614D
32-bit hash: 0x23D92F43
32-bit hash: 0xA1D0BE63
32-bit hash: 0xB0BF66B6
32-bit hash: 0x968F7074
32-bit hash: 0x76F75FD1
32-bit hash: 0x76A51358
3
  • If you have 3 duplicate lines, what distances do you consider: 1-2, 1-3, 2-3? Feb 9, 2014 at 13:35
  • I guess I'd treat them as two separate duplicates. The distances considered would be between the first and the second, then the second and the third.
    – SamSong
    Feb 9, 2014 at 13:40
  • Well, you first said that you consider the distance to the first occurrence only. Feb 9, 2014 at 15:07

1 Answer 1

2

You can do that using GNU awk:

$ cat a.txt 
string1
string2
string2
string1
string3

$ cat test.awk
{
    if($0 in lines) {
        distance += NR - lines[$0];
        ++count;
    }
    else {
        lines[$0] = NR;
    }
}
END {
    print distance / count;
}

$ awk -f test.awk < a.txt 
2

The above gives you the distance between the first occurrence of the line with the others. If you would like a distance between the next and the previous occurrences of the same line do:

    # ...
    if($0 in lines) {
        distance += NR - lines[$0];
        lines[$0] = NR; # <--- add this
        ++count;
    }
    # ...
4
  • I don't believe this works. I tested it with a file of the normal size but with only 2 duplicate lines (each repeated once) and manually calculated it.
    – SamSong
    Feb 9, 2014 at 14:50
  • @SamSong It works on the sample you provided. Post another sample. Feb 9, 2014 at 14:53
  • Edited the question with a sample from the input. Is there a way to post the entire file?
    – SamSong
    Feb 9, 2014 at 15:04
  • You Sir, have been great help. Thank you.
    – SamSong
    Feb 9, 2014 at 15:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.