1

I have a folder of 5000+ images in jpeg/png etc. How can I check if any of the images are the same. The images were collected through web scraping and have been sequentially renamed so I cannot compare file names.

I am currently checking if the hashes are the same however this is a very long process. I am currently using:

def sameIm(file_name1,file_name2):
    hash = imagehash.average_hash(Image.open(path + file_name1))
    otherhash = imagehash.average_hash(Image.open(path + file_name2))

    return (hash == otherhash)

Then nested loops. Comparing 1 image to 5000+ others takes about 5mins so comparing each to each would take days to compute.

Is there a faster way to do this in python. I was thinking parallel processing but would that still take a long time?

or is there another way to compare files which is faster?

Thanks

3
  • Do you want to check if they all start with the same name? Can you post some of the file names here? Jul 12, 2017 at 15:24
  • Why not hash all images by imagehash.average_hash to get a hash list, and then find duplicates in the hash list.
    – Huang
    Jul 12, 2017 at 15:29
  • You could first check the size of the file and only calculate the hash if they are equal..
    – f p
    Jul 12, 2017 at 15:30

4 Answers 4

3

There is indeed a much faster way of doing this:

import collections
import glob
import os


def dupDetector(dirpath, ext):
    hashes = collections.defaultdict(list)
    for fpath in glob.glob(os.path.join(dirpath, "*.{}".format(ext))):
        h = imagehash.average_hash(Image.open(fpath))
        hashes[h].append(fpath)

    for h,fpaths in hashes.items():
        if len(fpaths) == 1:
            print(fpaths[0], "is one of a kind")
            continue
        print("The following files are duplicates of each other (with the hash {}): \n\t{}".format(h, '\n\t'.join(fpaths)))

Using the dictionary with the file hash as a key gives you O(1) lookups, which means you don't need to do the pair-wise comparisons. You therefore go from a quadratic runtime, to a linear runtime (yay!)

5
  • Thanks! I have tried the code but am getting an invalid syntax on t.join on the last line
    – Stig
    Jul 12, 2017 at 15:59
  • @Stig: sorry, that was a typo from a muscle-memory-induced finger-twitch. It's fixed now Jul 12, 2017 at 17:01
  • dirpath vs dirpapth and missing brackets
    – ihsinme
    May 19, 2022 at 9:01
  • @ihsinme: could you please fix the missing brackets? I don't see it :( May 19, 2022 at 19:26
  • for fpath in glob.glob(os.path.join(dirpath, "*.{}".format(ext)):
    – ihsinme
    May 21, 2022 at 9:04
3

Why not compute hash only once?

hashes = [imagehash.average_hash(Image.open(path + fn)) for fn in file_names]
def compare_hashes(hash1, hash2):
    return hash1 == hash2
2

One solution is to keep using the hash but stocking it in a list of tuple (or a dic, i don't know wich is more efficient here) where the first element is the name of the image and the second is the hash. It should take aproximatively the same 5 mins.

If you have 5000 images, You compare the value of the first element of the list to the 4999 others

Then the second to the 4998 others (as you already checked the first one)

Then the third ...

This "just" make you do n²/2 comparisons (where n is the number of images)

2

Just use map structure to calculate hashes for each image,then store hashes as a key and name of the image as a value. As a result you would have unique images names array.

def get_hash(filename):
    return imagehash.average_hash(Image.open(path + filename))

def get_unique_images(filenames):
    hashes = {}
    for filename in filenames:
        image_hash = get_hash(filename)
        hashes[image_hash] = filename
    return hashes.values()

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.