1

I'm new to TensorFlow and every time I trained my model, it ends with the error :

RuntimeError: Coordinator stopped with threads still running: QueueRunnerThread-dummy_queue-sync_token_q_EnqueueMany

Does someone have any idea to fix it ?

I'm on Ubuntu 18.04 and using Python3.6 and TensorFlow 1.12.0 version, and using the ssd_mobilenet_v1_fpn_shared_box_predictor as model. To train it, I used the legacy/train.py script.

I've try to look at GitHub issues, but I don't understand many things. This error comes from : /home/ant/.local/lib/python3.6/site-packages/tensorflow/python/training/coordinator.py

It happens just after TensorFlow save the checkpoint

Here are logs when it crashes:

INFO:tensorflow:Stopping Training.
I0412 11:09:09.962124 139983862277952 tf_logging.py:115] Stopping Training.
INFO:tensorflow:Finished training! Saving model to disk.
I0412 11:09:09.962288 139983862277952 tf_logging.py:115] Finished training! Saving model to disk.
Traceback (most recent call last):
  File "object_detection/train.py", line 184, in <module>
    tf.app.run()
  File "/home/ant/.local/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "/home/ant/.local/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 306, in new_func
    return func(*args, **kwargs)
  File "object_detection/train.py", line 180, in main
    graph_hook_fn=graph_rewriter_fn)
  File "/usr/local/lib/python3.6/dist-packages/object_detection-0.1-py3.6.egg/object_detection/legacy/trainer.py", line 415, in train
    saver=saver)
  File "/home/ant/.local/lib/python3.6/site-packages/tensorflow/contrib/slim/python/slim/learning.py", line 785, in train
    ignore_live_threads=ignore_live_threads)
  File "/home/ant/.local/lib/python3.6/site-packages/tensorflow/python/training/supervisor.py", line 832, in stop
    ignore_live_threads=ignore_live_threads)
  File "/home/ant/.local/lib/python3.6/site-packages/tensorflow/python/training/coordinator.py", line 397, in join
    " ".join(stragglers))
RuntimeError: Coordinator stopped with threads still running: QueueRunnerThread-dummy_queue-sync_token_q_EnqueueMany
1
  • Well, it seems I've temporarily fix it: I just deleted the line NUM_STEPS: in my pipeline config file. It now save me a checkpoint every 25 steps. Apr 12, 2019 at 13:30

1 Answer 1

0

I have the same problem,my tf version is 1.13,then I upgraded to version 1.14,there's also a same error.

Finally, I change ignore_liver_threads=False to ignore_liver_threads=True in both ../tensorflow/contrib/slim/python/slim/learning.py and ../tensorflow/python/training/supervisor.py.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.