34

Is it possible to get the file names that were loaded using flow_from_directory ? I have :

datagen = ImageDataGenerator(
    rotation_range=3,
#     featurewise_std_normalization=True,
    fill_mode='nearest',
    width_shift_range=0.2,
    height_shift_range=0.2,
    horizontal_flip=True
)

train_generator = datagen.flow_from_directory(
        path+'/train',
        target_size=(224, 224),
        batch_size=batch_size,)

I have a custom generator for my multi output model like:

a = np.arange(8).reshape(2, 4)
# print(a)

print(train_generator.filenames)

def generate():
    while 1:
        x,y = train_generator.next()
        yield [x] ,[a,y]

Node that at the moment I am generating random numbers for a but for real training , I wish to load up a json file that contains the bounding box coordinates for my images. For that I will need to get the file names that were generated using train_generator.next() method. After I have that , I can load the file, parse the json and pass it instead of a. It is also necessary that the ordering of the x variable and the list of the file names that I get is the same.

2
  • Using only default Keras - it's not possible. But you can change a Keras code in order to do that. Commented Jan 18, 2017 at 21:03
  • Have you read my answer? Commented Feb 22, 2017 at 19:29

6 Answers 6

32

Yes is it possible, at least with version 2.0.4 (don't know about earlier version).

The instance of ImageDataGenerator().flow_from_directory(...) has an attribute with filenames which is a list of all the files in the order the generator yields them and also an attribute batch_index. So you can do it like this:

datagen = ImageDataGenerator()
gen = datagen.flow_from_directory(...)

And every iteration on generator you can get the corresponding filenames like this:

for i in gen:
    idx = (gen.batch_index - 1) * gen.batch_size
    print(gen.filenames[idx : idx + gen.batch_size])

This will give you the filenames of the images in the current batch.

3
  • 16
    It has to be noted, that this does not work if shuffle is True (default). You will always get the filenames in the order they are first processed, not neccesarily in the order they are returned from the generator.
    – Alex Guth
    Commented Oct 17, 2017 at 11:32
  • 12
    @AlexGuth what should one do when using shuffle=True?
    – Jash Shah
    Commented Feb 23, 2018 at 11:51
  • 3
    Generator call on last batch resets batch_index to 0. So you'll get idx = -1, which filters out last batch completely.
    – apatsekin
    Commented Mar 27, 2019 at 21:22
9

You can make a pretty minimal subclass that returns the image, file_path tuple by inheriting the DirectoryIterator:

import numpy as np
from keras.preprocessing.image import ImageDataGenerator, DirectoryIterator

class ImageWithNames(DirectoryIterator):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.filenames_np = np.array(self.filepaths)
        self.class_mode = None # so that we only get the images back

    def _get_batches_of_transformed_samples(self, index_array):
        return (super()._get_batches_of_transformed_samples(index_array),
                self.filenames_np[index_array])

In the init, I added a attribute that is the numpy version of self.filepaths so that we can easily index into that array to get the paths on each batch generation.

The only other change to the base class is to return a tuple that is the image batch super()._get_batches_of_transformed_samples(index_array) and the file paths self.filenames_np[index_array].

With that, you can make your generator like so:

imagegen = ImageDataGenerator()
datagen = ImageWithNames('/data/path', imagegen, target_size=(224,224))

And then check with

next(datagen)
2
  • Excellent answer. A couple small suggestions: the example classname doesn't match, should be "ImageWithNames". The example might also include subset="validation", shuffle=False in case it's not clear to people those should go here. Lastly, for those using keras from tensorflow the import would be from tensorflow.keras.preprocessing.... And maybe for the check data_batch, filenames = next(datagen), in case it's not super obvious. Commented Jul 3, 2019 at 16:00
  • 1
    This one is the right (or more pythonic) way of doing, IMO. Thanks!
    – iedmrc
    Commented May 8, 2020 at 14:49
5

at least with version 2.2.4,you can do it like this

datagen = ImageDataGenerator()
gen = datagen.flow_from_directory(...)
for file in gen.filenames:
    print(file)

or get the file path

for filepath in gen.filepaths:
    print(filepath)
1
  • 1
    This solution suffers from above mentioned shuffle=True mismatch of filenames and files in the batch.
    – Serge
    Commented Jul 27, 2020 at 17:05
2

Here is an example that works with shuffle=True as well. And also properly handles last batch. To make one pass:

datagen = ImageDataGenerator().flow_from_directory(...)    
batches_per_epoch = datagen.samples // datagen.batch_size + (datagen.samples % datagen.batch_size > 0)
for i in range(batches_per_epoch):
    batch = next(datagen)
    current_index = ((datagen.batch_index-1) * datagen.batch_size)
    if current_index < 0:
        if datagen.samples % datagen.batch_size > 0:
            current_index = max(0,datagen.samples - datagen.samples % datagen.batch_size)
        else:
            current_index = max(0,datagen.samples - datagen.batch_size)
    index_array = datagen.index_array[current_index:current_index + datagen.batch_size].tolist()
    img_paths = [datagen.filepaths[idx] for idx in index_array]
    #batch[0] - x, batch[1] - y, img_paths - absolute path
1

the below code might help. Overriding the flow_from_directory

    class AugmentingDataGenerator(ImageDataGenerator):
    def flow_from_directory(self, directory, mask_generator, *args, **kwargs):
        generator = super().flow_from_directory(directory, class_mode=None, *args, **kwargs)        
        seed = None if 'seed' not in kwargs else kwargs['seed']
        while True:           
            for image_path in generator.filepaths:
                # Get augmentend image samples
                image = next(generator)
                # print(image_path )

                yield image,image_path

# Create training generator
train_datagen = AugmentingDataGenerator(  
    rotation_range=10,
    width_shift_range=0.1,
    height_shift_range=0.1,
    rescale=1./255,
    horizontal_flip=True
)
train_generator = train_datagen.flow_from_directory(
    TRAIN_DIRECTORY_PATH, 
    target_size=(256, 256),
    shuffle = False,
    batch_size=BATCH_SIZE
)

# Create testing generator
test_datagen = AugmentingDataGenerator(rescale=1./255)
test_generator = test_datagen.flow_from_directory(
    TEST_DIRECTORY_PATH,  
    target_size=(256, 256),
    shuffle = False, # inorder to get imagepath of the same image
    batch_size=BATCH_SIZE 
)

And to check your images and file path returned

image,file_path = next(test_generator)
# print(file_path)
# plt.imshow(image)
0

I needed exactly this and I developed a simple function that works with shuffle=True or shuffle=False.

def get_indices_from_keras_generator(gen, batch_size):
    """
    Given a keras data generator, it returns the indices and the filepaths
    corresponding the current batch. 
    :param gen: keras generator.
    :param batch_size: size of the last batch generated.
    :return: tuple with indices and filenames
    """

    idx_left = (gen.batch_index - 1) * batch_size
    idx_right = idx_left + gen.batch_size if idx_left >= 0 else None
    indices = gen.index_array[idx_left:idx_right]
    filenames = [gen.filenames[i] for i in indices]
    return indices, filenames

Then, you would use it as follows:

for x, y in gen:
    indices, filenames = get_indices_from_keras_generator(gen)
1
  • you would need to supply a batch_size when calling it. something like: for x, y in gen: indices, filenames = get_indices_from_keras_generator(gen, gen.batch_size)
    – Createdd
    Commented Dec 20, 2021 at 17:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.