5

In my experiment, I want to train my custom model on imagenet datasets. For simplicity, I am interested 10/100 class classification task. But, direct downloading imagenet dataset from tfds requires a lot of space on a hard disk. Is there any workaround we could subset imagenet dataset so the subsetted imagenet dataset could fit for 10/100 class classification task? Does anyone know any way of making this happen? any idea?

In general, cifar10, cifar100 is quite handy to work with functional api of TensorFlow. But, in my experiment, I want to train my own model on imagenet. I want to avoid download imagenet dataset directly, instead, I want something less computational approach so I can train my custom model on subsetted imagenet (10 or 100 class classification). is there any way around to do this? any thoughts?

my attempt to download imagenet

this is my attempt to download imagenet dataset locally, then train my custom model on imagenet dataset. But it is time-consuming to download and load data for training. But this is what I did:

import keras
import tensorflow as tf
import tensorflow_datasets as tfds

## fetch imagenet dataset directly
imagenet = tfds.image.Imagenet2012()

## describe the dataset with DatasetInfo
C = imagenet.info.features['label'].num_classes
n_train = imagenet.info.splits['train'].num_examples
n_validation = imagenet.info.splits['validation'].num_examples

assert C == 1000
assert n_train == 1281167
assert n_validation == 50000

imagenet.download_and_prepare()   ## need more space in harddrive

# load imagenet data from disk as tf.data.Datasets
datasets = imagenet.as_dataset()
train_data, validation_data= datasets['train'], datasets['validation']
assert isinstance(train_data, tf.data.Dataset)
assert isinstance(validation_data, tf.data.Dataset)

If I do like this, this is time-consuming to download and needs more space on a hard-drive. Is there any easier way to subset imagenet dataset and get it from TensorFlow? Does anyone know an easier way of getting a smaller imagenet dataset for 10/100 classification task? any thoughts?

desired output

usually we can get cifar10, cifar100 from tf.keras.datasets. Can we subset the imagenet dataset to something range to (200k ~ 500K)? Is there any less painful approach to get imagenet dataset for training a custom model on imagenet data? any idea?

1 Answer 1

4

I figured it out by myself. I need to use tiny_imagenet_200:

import os, sys, wget
from zipfile import ZipFile

url = 'http://cs231n.stanford.edu/tiny-imagenet-200.zip'
tiny_imgdataset = wget.download('http://cs231n.stanford.edu/tiny-imagenet-200.zip', out = os.getcwd())
for file in os.listdir(os.getcwd()):
    if file.endswith(".zip"):
        zip = ZipFile(file)
        zip.extractall()
    else:
        print("not found")

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.