73

I'm testing different hyperparameters for a cnn model I built, but I'm having a small annoyance when viewing the summaries in Tensorboard. The problem seems to be that the data is just "added" in consecutive runs, so the functions result in a weird superposition unless I see the information as "relative" instead of "by step". See here:

X Type: Step

X Type: Relative

I've tried killing tensorboard's process and erasing the log files, but it seems it is not enough.

So the question is, how do I reset this information?

Thanks!!

2
  • it seems strange that even after deleting the log files and restarting the web app it still shows the old data. Maybe is the browser's caching ?
    – fabrizioM
    Dec 25, 2015 at 17:41
  • @fabrizioM This is still an issue. It usually requires multiple restarts, deleting cookies and using the default URL (e.g. localhost:6006 without any additional paths after that) to "fix" this behaviour. Since I am using Tensorboard in PyTorch and observe the same behaviour this definitely shows that it's an issue with Tensorboard itself. Feb 23, 2022 at 6:01

8 Answers 8

44

Note: The solution you've posted (erase TensorBoard's log files and kill the process) will work, but it isn't preferred, because it destroys historical information about your training.

Instead, you can have each new training job write to a new subdirectory (of your top-level log directory). Then, TensorBoard will consider each job a new "run" and will create a nice comparison view so you can see how the training differed between iterations of your model.

In the following an example from https://www.tensorflow.org/tensorboard/get_started:

model = create_model()
...
model.compile(...)

log_dir = "logs/fit/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)

model.fit(..., callbacks=[tensorboard_callback])
4
  • 9
    Indeed. What you describe is a killing feature of tensorflow that I've used a lot. But sometimes you just need to check repeatedly if everything behaves as you expect. That's why I needed to wipe everything and start from scratch
    – mathetes
    May 24, 2016 at 11:48
  • 3
    The link is broken :(
    – mjaskowski
    Apr 13, 2017 at 15:08
  • I'm having the same problem with binary_crossentropy and an autoencoder. My train values are lower than 1 but some of them are negatives. Can this be a problem? I have this: x_train.min()=-0.48 and x_train.max()=0.51 May 24, 2018 at 14:58
  • @mjaskowski, I think the link is working. April 2021. Apr 19, 2021 at 21:14
32

Yes, I believe ultimately this aspect is positive.
As an example, in my script I automate new run logs via datetime:

from datetime import datetime
now = datetime.now()
logdir = "tf_logs/.../" + now.strftime("%Y%m%d-%H%M%S") + "/"

Then when running TensorBoard you can click the different runs on and off provided you ran TensorBoard in the parent directory.

If you know you don't care about a previous run and want it out of your life, then yes, you need to remove the event files and restart TensorBoard.

0
24

Ok, for some reason it didn't work before but now it did:

You must erase Tensorboard's log files AND kill its process

After killing the process run fuser 6006/tcp -k to free port 6006 (if you're in linux) and fire tensorboard again.

2
  • 2
    It seems that all you have to do is delete the old directory (or rename it) to get it to refresh, though it doesn't notice until the number of seconds you passed to --reload_interval has elapsed.
    – golvok
    Feb 26, 2017 at 2:53
  • 2
    And for mac it would be lsof -i tcp:6006 | grep -v PID | awk '{print $2}' | xargs kill
    – jwsmithers
    Jan 31, 2019 at 17:33
6

I had a similar problem, but with a duplication of computational graphs: they've just added in tensorboard when I called

writer.add_graph(graph=sess.graph)

In my case, it wasn't about log files but about Jupyter Notebook context.

I figured out that after multiple runs of a Jupyter cell with a Graph definition the graph hasn't been reset but appeared in the context as a duplicate, so I added

tf.reset_default_graph()

before the start of building a computational graph.

Hope it will help.

2
  • definitely the fast hack for jupyter notebooks.
    – vwvan
    Apr 9, 2018 at 7:14
  • This solution worked for me to solve the duplicate issue, thank you @Стас Цепа
    – 0xFK
    Jun 22, 2018 at 6:28
5

I just figured out the solution to this problem. Just put each Events.out file in a separate folder inside your log directory. And you will get a nice visualization in tensorboard with each run in a different color.

2

Add the following snippet to your code and it should automatically reset your tensorboard.

if tf.gfile.Exists(dirpath):
   tf.gfile.DeleteRecursively(dirpath) 

This will delete the previous logs.

3
  • This does not work fully-- you must also shut down and restart TB each time.
    – Novak
    May 22, 2017 at 0:03
  • @Novak how do you shutdown the TB? !kill processId? In my case the notebook.list() shows the process ids even after kill. How do I remove them? I have no log files in the directory either.
    – Mihai.Mehe
    Nov 12, 2020 at 20:20
  • same can be accomplished by using: !rm -rf dirpath like so !rm -rf /home/my_project/logs/*
    – Mihai.Mehe
    Nov 12, 2020 at 20:30
0

Had a similar issue, which threw an error: 'You must feed a value for placeholder tensor 'dense_input' with dtype float and shape'. This specific issue was discussed here. The suggestion that worked for me used:

from keras.backend import clear_session
# Before instantiating a tf.data.Dataset obj & before model creation, call:
clear_session()

Also make sure you're using unique names for each model's TensorBoard log_dir. Then to view all models, run TensorBoard from terminal at the parent dir of the logs files, I.e.:

tensorboard --logdir <projDir>/logs/tf_log
-1

This automatically deletes the log directory.

import shutil

shutil.rmtree('log_dir', ignore_errors=True)
1
  • 1
    Please consider writing some explanations about what this function does and why you think this is a correct solution. Also consider using code formatting ;-)
    – Fabien
    Jul 13, 2017 at 22:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.