15

I want to search a file duplicate by its hash. For performance purposes I want to know if there is a stored hash/checksum for each file in NTFS/FAT file systems. If there is, I don't have to compute them all to search my file.

If there is, how to access it using .NET?

If it helps, it will be JPEG files. Do they have a checksum?

3 Answers 3

10

There is no such thing.

4
  • 2
    Windows allows random writes to a file. Could you imagine the overhead if each write required recomputing the file's checksum? Sep 29, 2009 at 3:14
  • I imagine that at least EXE files have a checksum, as other types may have.
    – Jader Dias
    Sep 29, 2009 at 3:18
  • 4
    @MarkRansom You could imagine it to be computed only when it's requested, and cached somewhere, with the only thing happening each time the file is written being cache invalidation - far less costly than recomputing it each time Jan 14, 2013 at 11:23
  • 4
    Also, a hash for error detection doesn't need to be cryptographically secure: it's OK to use some kind of cyclical pattern like plain addition or XOR; something where a few changed blocks can be compensated for without recomputing the complete hash. Nov 1, 2013 at 23:44
8

Windows does not store a hash for each file. As Jader Dias suggests, there are checksums for EXE's and DLL's but these are not the droids you are looking for.

Note that even if you had such a hash, it still does not guarantee uniqueness. If you found two files with the same hash (and size) you would still have to then compare contents to determine if the files were truly the same.

JPEG files may have some checksums or hashes, but you probably cannot count on them either.

1
  • 1
    +1 for "Note that even if you had such a hash, it still does not guarantee uniqueness." ... although it's true that very small changes almost always result in a unique hash, users have a way of producing those magical edge-case conditions. Sep 29, 2009 at 6:48
2

Windows though does have search now & if I recall correctly you can write your own plugins for it (in other words, to index files in a custom way). Presumably, you could write a plugin for JPGs & then simply make search API calls to find files (after Windows does the indexing).

2
  • 1
    I think Windows indexes text (as filenames), not images.
    – Jader Dias
    Oct 2, 2009 at 2:16
  • 2
    From <a href="msdn.microsoft.com/en-us/library/…>: The content indexed is based on the file and data types supported through add-ins... filters included in Window Search support over 200 common types of data including ... plain-text files, HTML, and many more. Sure, while it only natively supports certain files, as it says, you can index anything with a custom plugin. Certainly search can index MP3s - JPGs would be no different.
    – Vitali
    Oct 5, 2009 at 0:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.