1

I'm training a tensorflow.contrib.seq2seq encoder-decoder model, and the training time per minibatch is monotonically increasing.

Step Number: 10 Elapsed time: 52.89215302467346 Loss: 1.0420862436294556 Metrics: {'accuracy': 0.22499999} Step Number: 20 Elapsed time: 60.28505992889404 Loss: 0.8007364869117737 Metrics: {'accuracy': 0.28} Step Number: 30 Elapsed time: 73.98479580879211 Loss: 0.7292348742485046 Metrics: {'accuracy': 0.34} Step Number: 40 Elapsed time: 82.99069213867188 Loss: 0.6843382120132446 Metrics: {'accuracy': 0.345} Step Number: 50 Elapsed time: 86.97363901138306 Loss: 0.6808319687843323 Metrics: {'accuracy': 0.38999999} Step Number: 60 Elapsed time: 106.96697807312012 Loss: 0.601255476474762 Metrics: {'accuracy': 0.44} Step Number: 70 Elapsed time: 124.17725801467896 Loss: 0.5971778035163879 Metrics: {'accuracy': 0.405} Step Number: 80 Elapsed time: 137.91252613067627 Loss: 0.596596896648407 Metrics: {'accuracy': 0.43000001} Step Number: 90 Elapsed time: 146.6834409236908 Loss: 0.5921837687492371 Metrics: {'accuracy': 0.42500001}

All my data are artificially generated and are sampled randomly, meaning that (in general) there should be no difference between minibatches early in training and minibatches later in training. Additionally, all my data have the same input sequence length and the same output sequence length. Why might my model take longer to train later minibatches?

I found this relevant post, but I'm not changing my computational graph during my training loop.

To show some code, let's start in main:

def main(_):
    x_minibatch, y_minibatch, y_lengths_minibatch = construct_data_pipeline()

    model = import_model()

    train(model=model, x_minibatch=x_minibatch, y_minibatch=y_minibatch, y_lengths_minibatch=y_lengths_minibatch)

```

My data is stored as SequenceExamples, one per TFRecord file. My construct_data_pipeline() function is defined as follows:

def construct_data_pipeline():
    # extract TFRecord filenames located in data directory
    tfrecord_filenames = []
    for dirpath, dirnames, filenames in os.walk(tf.app.flags.FLAGS.data_dir):
        for filename in filenames:
            if filename.endswith('.tfrecord'):
                tfrecord_filenames.append(os.path.join(dirpath, filename))

    # read and parse data from TFRecords into tensors
    x, y, x_len, y_len = construct_examples_queue(tfrecord_filenames)

    # group tensors into minibatches
    x_minibatch, y_minibatch, y_lengths_minibatch = construct_minibatches(x=x, y=y,
                                                                      y_len=y_len,
                                                                      x_len=x_len)

    return x_minibatch, y_minibatch, y_lengths_minibatch

Stepping into construct_examples_queue()

def construct_examples_queue(tfrecords_filenames):
    number_of_readers = tf.flags.FLAGS.number_of_readers

    with tf.name_scope('examples_queue'):
        key, example_serialized = tf.contrib.slim.parallel_reader.parallel_read(tfrecords_filenames,
                                                                            tf.TFRecordReader,
                                                                            num_readers=number_of_readers)

        x, y, x_len, y_len = parse_example(example_serialized)

        return x, y, x_len, y_len

I don't think I can show parse_example, since the data isn't my own. The main parts are that I specify what I expect the SequenceExample to contain and then call

    context_parsed, sequence_parsed = tf.parse_single_sequence_example(example_serialized,
                                                                   context_features=context_features,
                                                                   sequence_features=sequence_features)

Skipping ahead to how I construct minibatches, I use

def construct_minibatches(x, y, y_len, x_len,
                      bucket_boundaries=list(range(400, tf.app.flags.FLAGS.max_x_len, 100))):

    batch_size = tf.app.flags.FLAGS.batch_size

    with tf.name_scope('batch_examples_using_buckets'):
        _, outputs = tf.contrib.training.bucket_by_sequence_length(input_length=len_x,
                                                               tensors=[x, y, y_len],
                                                               batch_size=batch_size,
                                                               bucket_boundaries=bucket_boundaries,
                                                               dynamic_pad=True,
                                                               capacity=2 * batch_size,
                                                               allow_smaller_final_batch=True)

        x_minibatch = outputs[0]
        y_minibatch = outputs[1]
        y_lengths_minibatch = outputs[2]
        return x_minibatch, y_minibatch, y_lengths_minibatch

Note: I had to change some variable names for privacy issues. Hopefully I didn't make any mistakes.

8
  • 1
    silly question, but you're sure it's not elapsed time since the start of training? What generates "elapsed time"?
    – vega
    Commented Jun 22, 2017 at 20:30
  • The loss is steadily decreasing, too.
    – NRitH
    Commented Jun 22, 2017 at 20:34
  • Elapsed time is initialized as start_time = time.time(). Then, after training on 10 minibatches, I call print(time.time() - start_time) and then start_time = time.time(). Commented Jun 22, 2017 at 20:41
  • To be more precise, I actually call print('Step Number: {}\tElapsed time: {}\tLoss: {}\tMetrics: {}'.format(step_number + 1, time.time() - start_time, loss, metrics)) Commented Jun 22, 2017 at 20:42
  • It's impossible to answer this question without having more details, including the parts of the code where you feed in data and call session.run. Commented Jun 22, 2017 at 21:01

1 Answer 1

1

Credit to faddy-w for solving two of my problems simultaneously!

It turns out I was changing my computational graph without knowing it.

I was calling

sess.run([model.optimizer.minimize(model.loss), model.y_predicted_logits],
                                 feed_dict={model.x: x_values,
                                            model.y_actual: y_values,
                                            model.y_actual_lengths: y_lengths_values})

from within a loop, where

model.loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(labels=self.y_actual,
                                                                      logits=self.y_predicted_logits))

and

model.optimizer = tf.train.GradientDescentOptimizer(learning_rate=initial_learning_rate)

without knowing that optimizer.minimize() adds additional operations to my graph.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.