0

I am trying to modify an existing tensorflow code. First, a 2d matrix of words is converted to a dataset from a geneartor and by map_strings_to_ints function and converted into vocab index. Then the following function is called.

dataset = dataset.apply(tf.contrib.data.bucket_by_sequence_length(element_length_func=lambda d: tf.shape(d)[0],
                                                                     bucket_boundaries=bucket_boundaries,
                                                                     bucket_batch_sizes=bucket_batch_sizes,
                                                                     padded_shapes=dataset.output_shapes,
                                                                     padding_values=constants.PAD_VALUE))

where each of the dataset elements was an array of size [None, None] (i.e., 2d mat).

Now for each element, I like to add another sequence of text. So each element is a tuple of previous 2d mat and the corresponding sentence/sequence that is each of the new dataset elements is a tuple of ([None, None],[None]), then how can I modify the above function?

I tried

dataset = dataset.apply(tf.contrib.data.bucket_by_sequence_length(element_length_func=lambda d,t: tf.shape(d)[0],
                                                                     bucket_boundaries=bucket_boundaries,
                                                                     bucket_batch_sizes=bucket_batch_sizes,
                                                                     padded_shapes=dataset.output_shapes,
                                                                     padding_values=constants.PAD_VALUE))

and few other tricks but got

TypeError: If shallow structure is a sequence, input must also be a sequence. Input has type: <class ‘int’>

Note that, the dataset elements are words mapped into vocab index (i.e., int)

1 Answer 1

0

This should be helpful for you:

X = np.array([[[1,2,3],[4,5,6]],[[7,8,9], [1,2,3], [4,5,6], [7,8,9]], [[1,2,3], [4,5,6]]])
Y = np.array([0,1,0])

def elements_gen():
    for x,y in zip(X,Y):
        yield (x,y)

dataset = tf.data.Dataset.from_generator(generator=elements_gen, output_shapes=([None, None], []), output_types=(tf.int32, tf.int32))

dataset = dataset.apply(tf.contrib.data.bucket_by_sequence_length(element_length_fun =lambda x,y: tf.shape(x)[0], bucket_boundaries=[4,7], bucket_batch_sizes=[2,2,2], padding_values=(0,0)))
iterator = dataset.make_one_shot_iterator()
next_element = iterator.get_next()

The issue is just what the error says, because the structure you are padding is a sequence the value used to pad the structure must be a sequence as well.

1
  • I tried that. But in my case Y is a 2d array [[i,go], [total,value,is], [not,so,bad]] and I got that error
    – rizwan
    Mar 31, 2019 at 8:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.