474

Is there any simple way of generating (and checking) MD5 checksums of a list of files in Python? (I have a small program I'm working on, and I'd like to confirm the checksums of the files).

10
  • 4
    Why not just use md5sum?
    – kennytm
    Commented Aug 7, 2010 at 19:55
  • 121
    Keeping it in Python makes it easier to manage the cross-platform compatibility.
    – Alexander
    Commented Aug 7, 2010 at 20:00
  • 2
    @kennytm The link you provided says this in the second paragraph: "The underlying MD5 algorithm is no longer deemed secure" while describing md5sum. That is why security-conscious programmers should not use it in my opinion.
    – Debug255
    Commented Feb 12, 2018 at 8:54
  • 1
    @Debug255 Good and valid point. Both md5sum and the technique described in this SO question should be avoided - it's better to use SHA-2 or SHA-3, if possible: en.wikipedia.org/wiki/Secure_Hash_Algorithms Commented Sep 27, 2018 at 8:33
  • 3
    Might be worth mentioning there are still valid reasons to use md5 that are not affected by it's brokenness for security purposes. (eg checking for bit rot in a system that uses baked in md5 creation during archival)
    – Smock
    Commented Feb 5, 2020 at 10:08

9 Answers 9

667

You can use hashlib.md5()

Note that sometimes you won't be able to fit the whole file in memory. In that case, you'll have to read chunks of 4096 bytes sequentially and feed them to the md5 method:

import hashlib
def md5(fname):
    hash_md5 = hashlib.md5()
    with open(fname, "rb") as f:
        for chunk in iter(lambda: f.read(4096), b""):
            hash_md5.update(chunk)
    return hash_md5.hexdigest()

Note: hash_md5.hexdigest() will return the hex string representation for the digest, if you just need the packed bytes use return hash_md5.digest(), so you don't have to convert back.

4
  • How could I decode the hex string ? It differs from the output of what md5sum returns
    – alper
    Commented Nov 17, 2021 at 20:09
  • @alper no it doesn't -- sorry to put it so flippantly-sounding, but there is no way that md5 differs for the same input -- if you're reading binary (not line-ending-agnostic) input, then this algorithm is deterministic -- md5's famous problem is that it might FAIL TO DIFFER for two different inputs
    – rsandwick3
    Commented Oct 15, 2022 at 4:43
  • @rsandwick3 As I understand md5 formula may end up generate same output for the two different inputs ?
    – alper
    Commented Oct 15, 2022 at 10:29
  • 2
    yes: crypto.stackexchange.com/questions/1434/…
    – rsandwick3
    Commented Oct 18, 2022 at 18:59
336

There is a way that's pretty memory inefficient.

single file:

import hashlib
def file_as_bytes(file):
    with file:
        return file.read()

print hashlib.md5(file_as_bytes(open(full_path, 'rb'))).hexdigest()

list of files:

[(fname, hashlib.md5(file_as_bytes(open(fname, 'rb'))).digest()) for fname in fnamelst]

Recall though, that MD5 is known broken and should not be used for any purpose since vulnerability analysis can be really tricky, and analyzing any possible future use your code might be put to for security issues is impossible. IMHO, it should be flat out removed from the library so everybody who uses it is forced to update. So, here's what you should do instead:

[(fname, hashlib.sha256(file_as_bytes(open(fname, 'rb'))).digest()) for fname in fnamelst]

If you only want 128 bits worth of digest you can do .digest()[:16].

This will give you a list of tuples, each tuple containing the name of its file and its hash.

Again I strongly question your use of MD5. You should be at least using SHA1, and given recent flaws discovered in SHA1, probably not even that. Some people think that as long as you're not using MD5 for 'cryptographic' purposes, you're fine. But stuff has a tendency to end up being broader in scope than you initially expect, and your casual vulnerability analysis may prove completely flawed. It's best to just get in the habit of using the right algorithm out of the gate. It's just typing a different bunch of letters is all. It's not that hard.

Here is a way that is more complex, but memory efficient:

import hashlib

def hash_bytestr_iter(bytesiter, hasher, ashexstr=False):
    for block in bytesiter:
        hasher.update(block)
    return hasher.hexdigest() if ashexstr else hasher.digest()

def file_as_blockiter(afile, blocksize=65536):
    with afile:
        block = afile.read(blocksize)
        while len(block) > 0:
            yield block
            block = afile.read(blocksize)


[(fname, hash_bytestr_iter(file_as_blockiter(open(fname, 'rb')), hashlib.md5()))
    for fname in fnamelst]

And, again, since MD5 is broken and should not really ever be used anymore:

[(fname, hash_bytestr_iter(file_as_blockiter(open(fname, 'rb')), hashlib.sha256()))
    for fname in fnamelst]

Again, you can put [:16] after the call to hash_bytestr_iter(...) if you only want 128 bits worth of digest.

36
  • 80
    I'm only using MD5 to confirm the file isn't corrupted. I'm not so concerned about it being broken.
    – Alexander
    Commented Aug 7, 2010 at 20:03
  • 99
    @TheLifelessOne: And despite @Omnifarious scary warnings, that is perfectly good use of MD5. Commented Aug 7, 2010 at 20:09
  • 24
    @GregS, @TheLifelessOne - Yeah, and next thing you know someone finds a way to use this fact about your application to cause a file to be accepted as uncorrupted when it isn't the file you're expecting at all. No, I stand by my scary warnings. I think MD5 should be removed or come with deprecation warnings. Commented Aug 7, 2010 at 20:21
  • 25
    I used this solution but it uncorrectly gave the same hash for two different pdf files. The solution was to open the files by specifing binary mode, that is: [(fname, hashlib.md5(open(fname, 'rb').read()).hexdigest()) for fname in fnamelst] This is more related to the open function than md5 but I thought it might be useful to report it given the requirement for cross-platform compatibility stated above (see also: docs.python.org/2/tutorial/…).
    – BlueCoder
    Commented Feb 26, 2013 at 14:09
  • 11
    @Omnifarious Saying "remove MD5 from the Python library" or even just saying "add deprecation warning to Python library" is like saying "Python should not be used, if existing stuff requires MD5, please use something else". Explain security implications in docs, sure, but removal or even just deprecation is insane suggestion.
    – hyde
    Commented Apr 3, 2013 at 13:48
36

I'm clearly not adding anything fundamentally new, but added this answer before I was up to commenting status, plus the code regions make things more clear -- anyway, specifically to answer @Nemo's question from Omnifarious's answer:

I happened to be thinking about checksums a bit (came here looking for suggestions on block sizes, specifically), and have found that this method may be faster than you'd expect. Taking the fastest (but pretty typical) timeit.timeit or /usr/bin/time result from each of several methods of checksumming a file of approx. 11MB:

$ ./sum_methods.py
crc32_mmap(filename) 0.0241742134094
crc32_read(filename) 0.0219960212708
subprocess.check_output(['cksum', filename]) 0.0553209781647
md5sum_mmap(filename) 0.0286180973053
md5sum_read(filename) 0.0311000347137
subprocess.check_output(['md5sum', filename]) 0.0332629680634
$ time md5sum /tmp/test.data.300k
d3fe3d5d4c2460b5daacc30c6efbc77f  /tmp/test.data.300k

real    0m0.043s
user    0m0.032s
sys     0m0.010s
$ stat -c '%s' /tmp/test.data.300k
11890400

So, looks like both Python and /usr/bin/md5sum take about 30ms for an 11MB file. The relevant md5sum function (md5sum_read in the above listing) is pretty similar to Omnifarious's:

import hashlib
def md5sum(filename, blocksize=65536):
    hash = hashlib.md5()
    with open(filename, "rb") as f:
        for block in iter(lambda: f.read(blocksize), b""):
            hash.update(block)
    return hash.hexdigest()

Granted, these are from single runs (the mmap ones are always a smidge faster when at least a few dozen runs are made), and mine's usually got an extra f.read(blocksize) after the buffer is exhausted, but it's reasonably repeatable and shows that md5sum on the command line is not necessarily faster than a Python implementation...

EDIT: Sorry for the long delay, haven't looked at this in some time, but to answer @EdRandall's question, I'll write down an Adler32 implementation. However, I haven't run the benchmarks for it. It's basically the same as the CRC32 would have been: instead of the init, update, and digest calls, everything is a zlib.adler32() call:

import zlib
def adler32sum(filename, blocksize=65536):
    checksum = zlib.adler32("")
    with open(filename, "rb") as f:
        for block in iter(lambda: f.read(blocksize), b""):
            checksum = zlib.adler32(block, checksum)
    return checksum & 0xffffffff

Note that this must start off with the empty string, as Adler sums do indeed differ when starting from zero versus their sum for "", which is 1 -- CRC can start with 0 instead. The AND-ing is needed to make it a 32-bit unsigned integer, which ensures it returns the same value across Python versions.

2
36

In Python 3.8+, you can can use the assignment operator := (along with hashlib) like this:

import hashlib
with open("your_filename.txt", "rb") as f:
    file_hash = hashlib.md5()
    while chunk := f.read(8192):
        file_hash.update(chunk)

print(file_hash.digest())
print(file_hash.hexdigest())  # to get a printable str instead of bytes

Consider using hashlib.blake2b instead of md5 (just replace md5 with blake2b in the above snippet). It's cryptographically secure and faster than MD5.

1
  • not sure, testing test = b"hello" with blake2b(test).hexdigest() and md5(test).hexdigest() is giving different results
    – Ricky Levi
    Commented 2 days ago
25
hashlib.md5(pathlib.Path('path/to/file').read_bytes()).hexdigest()
3
  • 5
    Hi! Please add some explanation to your code as to why this is a solution to the problem. Furthermore, this post is pretty old, so you should also add some information as to why your solution adds something that the others have not already addressed.
    – d_kennetz
    Commented Apr 24, 2019 at 14:17
  • 8
    It's another memory inefficient way Commented Aug 21, 2019 at 22:44
  • 4
    One-line solution. Perfect for a couple of tests! Commented Jun 26, 2020 at 8:18
18

In Python 3.11+, there's a new readable and memory-efficient method:

import hashlib
with open(path, "rb") as f:
    digest = hashlib.file_digest(f, "md5")
print(digest.hexdigest())
3

You could use simple-file-checksum1, which just uses subprocess to call openssl for macOS/Linux and CertUtil for Windows and extracts only the digest from the output:

Installation:

pip install simple-file-checksum

Usage:

>>> from simple_file_checksum import get_checksum
>>> get_checksum("path/to/file.txt")
'9e107d9d372bb6826bd81d3542a419d6'
>>> get_checksum("path/to/file.txt", algorithm="MD5")
'9e107d9d372bb6826bd81d3542a419d6'

The SHA1, SHA256, SHA384, and SHA512 algorithms are also supported.


1 Disclosure: I am the author of simple-file-checksum.

0

change the file_path to your file

import hashlib
def getMd5(file_path):
    m = hashlib.md5()
    with open(file_path,'rb') as f:
        lines = f.read()
        m.update(lines)
    md5code = m.hexdigest()
    return md5code
0
-1

you can make use of the shell here.

from subprocess import check_output

#for windows & linux
hash = check_output(args='md5sum imp_file.txt', shell=True).decode().split(' ')[0]

#for mac
hash = check_output(args='md5 imp_file.txt', shell=True).decode().split('=')[1]

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.