I'm training a tensorflow.contrib.seq2seq
encoder-decoder model, and the training time per minibatch is monotonically increasing.
Step Number: 10 Elapsed time: 52.89215302467346 Loss: 1.0420862436294556 Metrics: {'accuracy': 0.22499999}
Step Number: 20 Elapsed time: 60.28505992889404 Loss: 0.8007364869117737 Metrics: {'accuracy': 0.28}
Step Number: 30 Elapsed time: 73.98479580879211 Loss: 0.7292348742485046 Metrics: {'accuracy': 0.34}
Step Number: 40 Elapsed time: 82.99069213867188 Loss: 0.6843382120132446 Metrics: {'accuracy': 0.345}
Step Number: 50 Elapsed time: 86.97363901138306 Loss: 0.6808319687843323 Metrics: {'accuracy': 0.38999999}
Step Number: 60 Elapsed time: 106.96697807312012 Loss: 0.601255476474762 Metrics: {'accuracy': 0.44}
Step Number: 70 Elapsed time: 124.17725801467896 Loss: 0.5971778035163879 Metrics: {'accuracy': 0.405}
Step Number: 80 Elapsed time: 137.91252613067627 Loss: 0.596596896648407 Metrics: {'accuracy': 0.43000001}
Step Number: 90 Elapsed time: 146.6834409236908 Loss: 0.5921837687492371 Metrics: {'accuracy': 0.42500001}
All my data are artificially generated and are sampled randomly, meaning that (in general) there should be no difference between minibatches early in training and minibatches later in training. Additionally, all my data have the same input sequence length and the same output sequence length. Why might my model take longer to train later minibatches?
I found this relevant post, but I'm not changing my computational graph during my training loop.
To show some code, let's start in main
:
def main(_):
x_minibatch, y_minibatch, y_lengths_minibatch = construct_data_pipeline()
model = import_model()
train(model=model, x_minibatch=x_minibatch, y_minibatch=y_minibatch, y_lengths_minibatch=y_lengths_minibatch)
```
My data is stored as SequenceExample
s, one per TFRecord
file. My construct_data_pipeline()
function is defined as follows:
def construct_data_pipeline():
# extract TFRecord filenames located in data directory
tfrecord_filenames = []
for dirpath, dirnames, filenames in os.walk(tf.app.flags.FLAGS.data_dir):
for filename in filenames:
if filename.endswith('.tfrecord'):
tfrecord_filenames.append(os.path.join(dirpath, filename))
# read and parse data from TFRecords into tensors
x, y, x_len, y_len = construct_examples_queue(tfrecord_filenames)
# group tensors into minibatches
x_minibatch, y_minibatch, y_lengths_minibatch = construct_minibatches(x=x, y=y,
y_len=y_len,
x_len=x_len)
return x_minibatch, y_minibatch, y_lengths_minibatch
Stepping into construct_examples_queue()
def construct_examples_queue(tfrecords_filenames):
number_of_readers = tf.flags.FLAGS.number_of_readers
with tf.name_scope('examples_queue'):
key, example_serialized = tf.contrib.slim.parallel_reader.parallel_read(tfrecords_filenames,
tf.TFRecordReader,
num_readers=number_of_readers)
x, y, x_len, y_len = parse_example(example_serialized)
return x, y, x_len, y_len
I don't think I can show parse_example
, since the data isn't my own. The main parts are that I specify what I expect the SequenceExample
to contain and then call
context_parsed, sequence_parsed = tf.parse_single_sequence_example(example_serialized,
context_features=context_features,
sequence_features=sequence_features)
Skipping ahead to how I construct minibatches, I use
def construct_minibatches(x, y, y_len, x_len,
bucket_boundaries=list(range(400, tf.app.flags.FLAGS.max_x_len, 100))):
batch_size = tf.app.flags.FLAGS.batch_size
with tf.name_scope('batch_examples_using_buckets'):
_, outputs = tf.contrib.training.bucket_by_sequence_length(input_length=len_x,
tensors=[x, y, y_len],
batch_size=batch_size,
bucket_boundaries=bucket_boundaries,
dynamic_pad=True,
capacity=2 * batch_size,
allow_smaller_final_batch=True)
x_minibatch = outputs[0]
y_minibatch = outputs[1]
y_lengths_minibatch = outputs[2]
return x_minibatch, y_minibatch, y_lengths_minibatch
Note: I had to change some variable names for privacy issues. Hopefully I didn't make any mistakes.
start_time = time.time()
. Then, after training on 10 minibatches, I callprint(time.time() - start_time)
and thenstart_time = time.time()
.print('Step Number: {}\tElapsed time: {}\tLoss: {}\tMetrics: {}'.format(step_number + 1, time.time() - start_time, loss, metrics))