In my experiment, I want to train my custom model on imagenet
datasets. For simplicity, I am interested 10/100 class classification task. But, direct downloading imagenet
dataset from tfds
requires a lot of space on a hard disk. Is there any workaround we could subset imagenet
dataset so the subsetted imagenet
dataset could fit for 10/100 class classification task? Does anyone know any way of making this happen? any idea?
In general, cifar10
, cifar100
is quite handy to work with functional api of TensorFlow. But, in my experiment, I want to train my own model on imagenet
. I want to avoid download imagenet
dataset directly, instead, I want something less computational approach so I can train my custom model on subsetted imagenet
(10 or 100 class classification). is there any way around to do this? any thoughts?
my attempt to download imagenet
this is my attempt to download imagenet
dataset locally, then train my custom model on imagenet
dataset. But it is time-consuming to download and load data for training. But this is what I did:
import keras
import tensorflow as tf
import tensorflow_datasets as tfds
## fetch imagenet dataset directly
imagenet = tfds.image.Imagenet2012()
## describe the dataset with DatasetInfo
C = imagenet.info.features['label'].num_classes
n_train = imagenet.info.splits['train'].num_examples
n_validation = imagenet.info.splits['validation'].num_examples
assert C == 1000
assert n_train == 1281167
assert n_validation == 50000
imagenet.download_and_prepare() ## need more space in harddrive
# load imagenet data from disk as tf.data.Datasets
datasets = imagenet.as_dataset()
train_data, validation_data= datasets['train'], datasets['validation']
assert isinstance(train_data, tf.data.Dataset)
assert isinstance(validation_data, tf.data.Dataset)
If I do like this, this is time-consuming to download and needs more space on a hard-drive. Is there any easier way to subset imagenet
dataset and get it from TensorFlow? Does anyone know an easier way of getting a smaller imagenet
dataset for 10/100 classification task? any thoughts?
desired output
usually we can get cifar10
, cifar100
from tf.keras.datasets
. Can we subset the imagenet dataset to something range to (200k ~ 500K)? Is there any less painful approach to get imagenet dataset for training a custom model on imagenet
data? any idea?