Typical approaches recommend reading the binary via FileStream and comparing it byte-by-byte.
- Would a checksum comparison such as CRC be faster?
- Are there any .NET libraries that can generate a checksum for a file?
|
Typical approaches recommend reading the binary via FileStream and comparing it byte-by-byte.
|
|||||
|
|
A checksum comparison will be most likely be slower than a byte-by-byte comparison. In order to generate a checksum, you'll need to load each byte of the file, and perform processing on it. You'll then have to do this on the second file. The processing will almost definitely be slower than the comparison check. As for generating a checksum: You can do this easily with the cryptography classes. Here's a short example of generating an MD5 checksum with C#. However, a checksum may be faster and make more sense if you can pre-compute the checksum of the "test" or "base" case. If you have an existing file, and you're checking to see if a new file is the same as the existing one, pre-computing the checksum on your "existing" file would mean only needing to do the DiskIO one time, on the new file. This would likely be faster than a byte-by-byte comparison. |
|||||||||||||||||
|
|
The slowest possible method is to compare two files byte by byte. The fastest I've been able to come up with is a similar comparison, but instead of one byte at a time, you would use an array of bytes sized to Int64, and then compare the resulting numbers. Here's what I came up with:
In my testing, I was able to see this outperform a straightforward ReadByte() scenario by almost 3:1. Averaged over 1000 runs, I got this method at 1063ms, and the method below (straightforward byte by byte comparison) at 3031ms. Hashing always came back sub-second at around an average of 865ms. This testing was with an ~100MB video file. Here's the ReadByte and hashing methods I used, for comparison purposes:
|
|||||||||||||||||
|
|
In addition to Reed Copsey's answer:
For example, if the two files are of different length then you know they cannot be identical, and you don't even have to compare their actual content. |
|||||||||||||
|
|
The only thing that might make a checksum comparison slightly faster than a byte-by-byte comparison is the fact that you are reading one file at a time, somewhat reducing the seek time for the disk head. That slight gain may however very well be eaten up by the added time of calculating the hash. Also, a checksum comparison of course only has any chance of being faster if the files are identical. If they are not, a byte-by-byte comparison would end at the first difference, making it a lot faster. You should also consider that a hash code comparison only tells you that it's very likely that the files are identical. To be 100% certain you need to do a byte-by-byte comparison. If the hash code for example is 32 bits, you are about 99,99999998% certain that the files are identical if the hash codes match. That is close to 100%, but if you truly need 100% certainty, that's not it. |
|||||||||||||||||||||
|
|
It's getting even faster if you don't read in small 8 byte chunks but put a loop around, reading a larger chunk. I reduced the average comparison time to 1/4.
|
|||||||||
|
|
Edit: This method would not work for comparing binary files! In .NET 4.0, the
Which means you could use:
|
|||||||||||||
|
|
Honestly, I think you need to prune your search tree down as much as possible. Things to check before going byte-by-byte:
Also, reading large blocks at a time will be more efficient since drives read sequential bytes more quickly. Going byte-by-byte causes not only far more system calls, but it causes the read head of a traditional hard drive to seek back and forth more often if both files are on the same drive. Read chunk A and chunk B into a byte buffer, and compare them (do NOT use Array.Equals, see comments). Tune the size of the blocks until you hit what you feel is a good trade off between memory and performance. You could also multi-thread the comparison, but don't multi-thread the disk reads. |
|||||||||||||||||||||
|
|
My experiments show that it definitely helps to call Stream.ReadByte() fewer times, but using BitConverter to package bytes does not make much difference against comparing bytes in a byte array. So it is possible to replace that "Math.Ceiling and iterations" loop in the comment above with the simplest one:
I guess it has to do with the fact that BitConverter.ToInt64 needs to do a bit of work (check arguments and then perform the bit shifting) before you compare and that ends up being the same amount of work as compare 8 bytes in two arrays. |
|||||
|
|
If you do decide you truly need a full byte-by-byte (i.e., binary) comparison (see other answers for discussion of hashing) then the one-line solution is:
Unlike some other posted answers, this works correctly for any kind of file, text or binary. However, because this code images both files into memory entirely, it should only be used for small-to-medium file size scenarios. In fact, this could be an optimal solution for file sizes which expected to be less than 85K, since small allocations in For example, even though On the other hand, the above does not include eager abort for differently-sized files, which can provide a tangible performance difference. This is because file length is available in the
You could also extend this to avoid the secondary fetches if the |
||||
|
|
|
Another improvement on large files with identical length, might be to not read the files sequentially, but rather compare more or less random blocks. You can use multiple threads, starting on different positions in the file and comparing either forward or backwards. This way you can detect changes at the middle/end of the file, faster than you would get there using a sequential approach. |
|||||
|
|
If you only need to compare two files, I guess the fastest way would be (in C, I don't know if it's applicable to .NET)
OTOH, if you need to find if there are duplicate files in a set of N files, then the fastest way is undoubtedly using a hash to avoid N-way bit-by-bit comparisons. |
|||
|
|
|
Something (hopefully) reasonably efficient:
|
|||
|
|
|
If the files are not too big, you can use:
It will only be feasible to compare hashes if the hashes are useful to store. (Edited the code to something much cleaner.) |
|||
|
|
|
I think there are applications where "hash" is faster than comparing byte by byte. If you need to compare a file with others or have a thumbnail of a photo that can change. It depends on where and how it is using.
Here, you can get what is the fastest.
Optionally, we can save the hash in a database. Hope this can help |
|||
|
|