I have a simple question: How do I change the built-in Python logger's print
function to tqdm.write
such that logging messages do not interfere with tqdm's progress bars?
-
4If you have to stick with tqdm, then @RolKau's answer is the way to go, but tqdm clears and redraws all of the progress bars on each write, so it's easy to overwhelm it if you have a lot of output. If you're not suck with tqdm, Enlighten does what you need out of the box and will hold up better under load since it doesn't rely on redrawing.– avisoNov 12, 2018 at 20:35
6 Answers
tqdm
now has a built-in contextmanager for redirecting the logger:
import logging
from tqdm import trange
from tqdm.contrib.logging import logging_redirect_tqdm
LOG = logging.getLogger(__name__)
if __name__ == '__main__':
logging.basicConfig(level=logging.INFO)
with logging_redirect_tqdm():
for i in trange(9):
if i == 4:
LOG.info("console logging redirected to `tqdm.write()`")
# logging restored
Copied from tqdm documentation
-
3out of all the above answers. only this worked. Should be marked as accepted.– AniOct 12, 2021 at 19:46
-
1If you've set your own formatter for
LOG
, then you should dologging_redirect_tqdm(loggers=[LOG])
. This will causetqdm
to useLOG
's formatter. Jun 16, 2022 at 14:08 -
this answer is more up to date, works and is much more straightforward than the accepted one. Jul 22, 2022 at 15:47
-
It does require a context manager wherever one wants to use
tqdm.write
though, whereas the other solution can be used to make the change globally. Jun 27 at 4:22
You need a custom logging handler:
import logging
import tqdm
class TqdmLoggingHandler(logging.Handler):
def __init__(self, level=logging.NOTSET):
super().__init__(level)
def emit(self, record):
try:
msg = self.format(record)
tqdm.tqdm.write(msg)
self.flush()
except Exception:
self.handleError(record)
and then add this to the logging chain:
import time
log = logging.getLogger(__name__)
log.setLevel(logging.INFO)
log.addHandler(TqdmLoggingHandler())
for i in tqdm.tqdm(range(100)):
if i == 50:
log.info("Half-way there!")
time.sleep(0.1)
Edit: Fixed a bug in the call to super TqdmLoggingHandler's init method, that was pointed out by diligent reader @BlaineRogers in the comments. (If anyone wants to read further about this murky area of Python, I recommend https://fuhm.net/super-harmful/)
-
I needed something to use logging module for output, solved it like this: stackoverflow.com/a/41224909/638504. Output is like this:
2016-12-19 15:35:06 [INFO ] 16%|#####9 | 768/4928 [07:04<40:50, 1.70it/s]
– ddofborgDec 21, 2016 at 11:42 -
1it is important to use
import tqdm
notfrom tqdm import tqdm
else the IO will interrupt the progress bar Feb 13, 2017 at 15:28 -
super(self.__class__, self).__init__(level)
will cause aRecursionError
if you ever try to instantiate a subclass ofTqdmLoggingHandler
.super(cls, self).__init__()
calls the__init__
method of the class aftercls
in the method resolution order ofself.__class__
. LetMyTqdmLoggingHandler
be a subclass ofTqdmLoggingHandler
. ThenMyTqdmLoggingHandler()
callsTqdmLoggingHandler.__init__(self)
, which callssuper(MyTqdmLoggingHandler, self).__init__()
, which callsTqdmLoggingHandler.__init__(self)
. Feb 7, 2019 at 15:02 -
1@BlaineRogers You are absolutely right! Thank you for pointing this out. I must apparently have written this before I became familiar with the MRO. I will correct the answer, in case anyone wants to use the code.– RolKauFeb 11, 2019 at 17:51
-
I don't think that reference to the ancient "Super considered harmful" does anyone any good. Please see Raymond Hettinger's talk "Super Considered Super!" for a thorough rebuttal. youtu.be/EiOglTERPEo– traalDec 28, 2021 at 20:23
Based on RolKau's answer above, simplified:
import logging
from tqdm import tqdm
class TqdmLoggingHandler(logging.StreamHandler):
"""Avoid tqdm progress bar interruption by logger's output to console"""
# see logging.StreamHandler.eval method:
# https://github.com/python/cpython/blob/d2e2534751fd675c4d5d3adc208bf4fc984da7bf/Lib/logging/__init__.py#L1082-L1091
# and tqdm.write method:
# https://github.com/tqdm/tqdm/blob/f86104a1f30c38e6f80bfd8fb16d5fcde1e7749f/tqdm/std.py#L614-L620
def emit(self, record):
try:
msg = self.format(record)
tqdm.write(msg, end=self.terminator)
except RecursionError:
raise
except Exception:
self.handleError(record)
Testing:
import time
log = logging.getLogger(__name__)
log.setLevel(logging.INFO)
log.addHandler(TqdmLoggingHandler())
# ^-- Assumes this will be the unique handler emitting messages to sys.stdout.
# If other handlers output to sys.stdout (without tqdm.write),
# progress bar will be interrupted by those outputs
for i in tqdm(range(20)):
log.info(f"Looping {i}")
time.sleep(0.1)
Caveat: if you're working in a Jupyter notebook, progress bar will be interrupted and AFAIK there's no way to avoid it.
One easy solution, not so elegant, is convert the tqdm object to string. After that yo can log the message (or handle it as you want). The "format_dict" attribute may be useful as well:
from tqdm import tqdm
import time
#loop with progressbar:
it=tqdm(total=10)
for i in range(10):
time.sleep(0.1)
it.update(1)
it.close()
print("\n","--"*10)
# Convert tdqm object last output to sting
str_bar_msg = str(it)
print(str_bar_msg)
# See attributes:
print(it.format_dict)
output:
100%|██████████| 10/10 [00:01<00:00, 8.99it/s]
--------------------
100%|██████████| 10/10 [00:01<00:00, 8.98it/s]
{'n': 10, 'total': 10, 'elapsed': 1.1145293712615967, 'ncols': None, 'nrows': None, 'prefix': '', 'ascii': False, 'unit': 'it', 'unit_scale': False, 'rate': None, 'bar_format': None, 'postfix': None, 'unit_divisor': 1000, 'initial': 0, 'colour': None}
Best regards
-
Use
from tqdm.notebook import tqdm
if you are in Jupyter Notebook Jan 25, 2022 at 22:06 -
actually use
from tqdm.auto import tqdm
to auto detect if you are in a jupyter notebook and use the appropriate bar– tbrugereSep 1 at 18:40
the easiest way is change stream of StreamHandler
object e.g.:
import logging
from tqdm import tqdm, trange
import time
log = logging.getLogger(__name__)
log.setLevel(logging.INFO)
handler = logging.StreamHandler()
handler.setStream(tqdm) # <-- important
handler = log.addHandler(handler)
for i in trange(100):
if i == 50:
log.info("Half-way there!")
time.sleep(0.1)
-
Not working, a warning appears in
log.info
when executed directly in COLAB.– krenerdSep 15, 2021 at 12:34 -
replace
log.info("Half-way there!")
bytqdm.write("Half-way there!")
. result is the same, wchich mean the problem is module tqdm itself, because.write(...)
is official print method while using tqdm. Sep 16, 2021 at 3:21
The new io handler is useful!
class TqdmToLogger(io.StringIO):
logger = None
level = None
buf = ""
def __init__(self, logger, level=None, mininterval=5):
super(TqdmToLogger, self).__init__()
self.logger = logger
self.level = level or logging.INFO
self.mininterval = mininterval
self.last_time = 0
def write(self, buf):
self.buf = buf.strip("\r\n\t ")
def flush(self):
if len(self.buf) > 0 and time.time() - self.last_time > self.mininterval:
self.logger.log(self.level, self.buf)
self.last_time = time.time()```
# before this line, you need to create logger with file handler
tqdm_out = TqdmToLogger(logger)
tbar = tqdm(sample, total=len(sample), file=tqdm_out)
logger.info("Model Inference.")
for it, batch_data in enumerate(tbar):
pass
```
-
As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center.– Community BotNov 4, 2022 at 15:19