I have a long list of lists of integers (representing sentences, each one of different sizes) that I want to feed using the tf.data library. Each list (of the lists of list) has different length, and I get an error, which I can reproduce here:
t = [[4,2], [3,4,5]]
dataset = tf.data.Dataset.from_tensor_slices(t)
The error I get is:
ValueError: Argument must be a dense tensor: [[4, 2], [3, 4, 5]] - got shape [2], but wanted [2, 2].
Is there a way to do this?
EDIT 1: Just to be clear, I don't want to pad the input list of lists (it's a list of sentences containing over a million elements, with varying lengths) I want to use the tf.data library to feed, in a proper way, a list of lists with varying length.
tf.data.Dataset.from_tensor_slices
it should work, and you should then be able to transform each sentence to a list of integers usingdataset.map(your_function)
. You can then usedataset.padded_batch
to automatically add the padding.tf.data
, it uses queues in the background and only processes data as needed. You can "prefetch" data to make sure that your GPU is never waiting for data and is working at 100%. As data is consumed at one end (for training), the queues before get filled up with data. You can even have multiple workers withnum_parallel_calls
.